August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

•Download as PPTX, PDF•

0 likes•26,054 views

Big data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Upstream data sources can 'drift' due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail. StreamSets Data Collector (SDC) is an open source platform for building big data ingest pipelines that allows you to design, execute and monitor robust data flows. In this session we'll look at how SDC's "intent-driven" approach keeps the data flowing, whether you're processing data 'off-cluster', in Spark, or in MapReduce. StreamSets software delivers performance management for data flows that feed the next generation of big data applications. Its mission is to bring operational excellence to the management of data in motion, so that data arrives on time and with quality, accelerating analysis and decision making. StreamSets Data Collector is in use at hundreds of companies where it brings unprecedented visibility into and control over data as it moves between an expanding variety of sources and destinations. Speakers: Pat Patterson has been working with Internet technologies since 1997, building software and working with communities at Sun Microsystems, Huawei, Salesforce and StreamSets. At Sun, Pat was the community lead for the OpenSSO open source project, while at Huawei he developed cloud storage infrastructure software. Part of the developer evangelism team at Salesforce, Pat focused on identity, integration and the Internet of Things. Now community champion at StreamSets, Pat is responsible for the care and feeding of the StreamSets open source community.

Technology

Open Source Big Data Ingest with
StreamSets Data Collector
Pat Patterson
Community Champion
@metadaddy
pat@streamsets.com

Traditional and Big Data
Founders
Company Background
Top tier Investors
Momentum to Date
Strategic Partners
Launched 2014; exited stealth 9/15
~30 employees
Double-digit enterprise customers
10,000 downloads

Past ETL ETL
Emerging Ingest Analyze
Data Sources Data Stores Data Consumers
Market Trends

Data Drift
The unpredictable, unannounced and unending mutation of data characteristics caused by
the operation, maintenance and modernization of the systems that produce the data
Structure
Drift
Semantic
Drift
Infrastructure
Drift

Delayed and
False Insights
Solving Data Drift
Tools
Applications
Data Stores Data ConsumersData Sources
Poor Data QualityData Drift
Custom code
Fixed-schema

Trusted InsightsData KPIs
Solving Data Drift
Tools
Applications
Data Stores Data ConsumersData Sources
Data Drift
Intent-Driven
Drift-Handling

SQL on Hadoop (Hive) Y/Y Click Through Rate
80% of analyst time is spent preparing and validating data,
while the remaining 20% is actual data analysis
Example: Data Loss and Corrosion

StreamSets Data Collector
Open source software for the
rapid development and
reliably operation of complex
data flows.
➢ Efficiency
➢ Control
➢ Agility

SDC Demo
StreamSets
Data Collector
Apache Kafka
Apache Kudu
↘
↘

SF Bay Area Data Ingest Meetup - Aug 25, Palo Alto, CA
MapR Big Data Everywhere - Aug 30, San Francisco, CA
Strata + Hadoop World - Sep 27-29, New York, NY
Upcoming Events

Structure
Drift
Data structures and
formats evolve and
change unexpectedly
Implication:
Data Loss
Data Squandering
Delimited Data
107.3.137.195 fe80::21b:21ff:fe83:90fa
Attribute Format
Changes
{
“first“: “jon”
“last“: “smith”
“email“: “jsmith@acme.com”
“add1“: “123 Washington”
“add2“: “”
“city“: “Tucson”
“state“: “AZ”
“zip“: “85756”
}
{
“first“: “jane”
“last“: “smith”
“email“: “jane@earth.net”
“add1“: “456 Fillmore”
“add2“: “Apt 120”
“city“: “Fairfield”
“state“: “VA”
“zip“: “24435-1001”
“phone”: “401-555-1212”
}
Data Structure Evolution
Structure Drift

Semantic
Drift
Data semantics change
with evolving applications
Implication:
Data Corrosion
Data Loss
Semantic Drift
24122-52172 00-24122-52172
Account Number Expansion
M134: user {jsmith} read access granted {ac:24122-52172}
M134: user {jsmith} read access granted {ca.ac:24122-52172}
Namespace Qualification
……
…,3588310669797950,$91.41,jcb,K1088-W#9,…
…,6759006011936944,$155.04,switch,A6504-Y#9,…
…,6771111111151415,$37.78,laser,Q9936-T#9,…
…,3585905063294299,$164.48,jcb,S4643-H#9,…
…,5363527828638736,$117.52,mastercard,X3286-P#9,…
…,4903080150282806,$168.03,switch,I9133-W#3,…
……
Outlier / Anomaly Detection

Infrastructure
Drift
Physical and Logical
Infrastructure changes
rapidly
Implication:
Poor Agility
Operational Downtime
Data Center 1 Data Center 2 Data Center n
3rd Party Service Provider
App a App k
App q
Cloud
Infrastructure
Infrastructure Drift

Big data tools such as Hadoop and Spark allow you to process data at unprecedented scale, but keeping your processing engine fed can be a challenge. Metadata in upstream sources can ‘drift’ due to infrastructure, OS and application changes, causing ETL tools and hand-coded solutions to fail. StreamSets Data Collector (SDC) is an Apache 2.0 licensed open source platform for building big data ingest pipelines that allows you to design, execute and monitor robust data flows. In this session we’ll look at how SDC’s “intent-driven” approach keeps the data flowing, with a particular focus on clustered deployment with Spark and other exciting Spark integrations in the works.

Webinar: The Modern Streaming Data Stack with Kinetica & StreamSets

Kinetica

Enterprises are now faced with wrangling massive volumes of complex, streaming data from a variety of different sources, a new paradigm known as extreme data. However, the traditional data integration model that’s based on structured batch data and stable data movement patterns makes it difficult to analyze extreme data in real-time. Join Matt Hawkins, Principal Solutions Architect at Kinetica and Mark Brooks, Solution Engineer at StreamSets as they share how innovative organizations are modernizing their data stacks with StreamSets and Kinetica to enable faster data movement and analysis.In this webinar we’ll explore: The modern data architecture required for dealing with extreme data How StreamSets enables continuous data movement and transformation across the enterprise How Kinetica harnesses the power of GPUs to accelerate analytics on streaming data A live demo of StreamSets and Kinetica connector to enable high speed data ingestion, queries and data visualization

Building Custom Big Data Integrations

Pat Patterson

Big data ingest frameworks ship with an array of connectors for common data origins and destinations, such as flat files, S3, HDFS, Kafka etc, but sometimes, you need to send data to, or receive data from a system that's not on the list. StreamSets includes template code for building your own connectors and processors; we'll walk through the process of building a simple destination that sends data to a REST web service, and show how it can be extended to target more sophisticated systems such as Salesforce Wave Analytics.

Dealing with Drift: Building an Enterprise Data Lake

Pat Patterson

Presented at San Diego Predictive Analytics Meetup, Dec 5, 2017 Cox Automotive comprises more than 25 companies dealing with different aspects of the car ownership lifecycle, with data as the common language they all share. The challenge for Cox was to create an efficient engine for the timely and trustworthy ingest of data capability for an unknown but large number of data assets from practically any source. Discover how their big data engineering team overcame data drift and are now populating a data lake, allowing analysts easy access to data from their subsidiary companies and producing new data assets unique to the industry.

Enabling Next Gen Analytics with Azure Data Lake and StreamSets

Streamsets Inc.

Big data and the cloud are perfect partners for companies who want to unlock maximum value from all of their unstructured, semi-structured, and structured data. The challenge has been how to create and manage a reliable end-to-end solution that spans data ingestion, storage and analysis in the face of the volume, velocity and variety of big data sources. In this webinar, we will show you how to achieve big data bliss by combining StreamSets Data Collector, which specializes in creating and running complex any-to-any dataflows, with Microsoft's Azure Data Lake and Azure analytic solutions. We will walk through an example of how a major bank is using StreamSets to transport their on-premise data to the Azure Cloud Computing Platform and Azure Data Lake to take advantage of analytics tools with unprecedented scale and performance.

Data Driving Yahoo Mail Growth and Evolution with a 50 PB Hadoop Warehouse

DataWorks Summit

Yahoo Mail has 200+ million users a month and generates hundreds of terabytes of data per day, which continues to grow steadily. The nature of email messages has also evolved: for example, today the majority of them are generated by machines, consisting of newsletters, social media notifications, purchase invoices, travel bookings, and the like, which drove innovations in product development to help users organize their inboxes. Since 2014, the Yahoo Mail Data Engineering team took on the task of revamping the Mail data warehouse and analytics infrastructure in order to drive the continued growth and evolution of Yahoo Mail. Along the way we have built a 50 PB Hadoop warehouse, and surrounding analytics and machine learning programs that have transformed the way data plays in Yahoo Mail. In this session we will share our experience from this 3 year journey, from the system architecture, analytics systems built, to the learnings from development and drive for adoption.

Case Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud

Rick Bilodeau

Presto is an open source distributed SQL query engine for running interactive analytic queries against data sources of all sizes ranging from gigabytes to petabytes. Presto was designed and written from the ground up for interactive analytics and approaches the speed of commercial data warehouses while scaling to the size of organizations like Facebook. One key feature in Presto is the ability to query data where it lives via a uniform ANSI SQL interface. Presto’s connector architecture creates an abstraction layer for anything that can be expressed in a row-like format, such as HDFS, Amazon S3, Azure Storage, NoSQL stores, relational databases, Kafka streams and even proprietary data stores. Furthermore, a single Presto query can combine data from multiple sources, allowing for analytics across your entire organization. This talk will be co-presented by Facebook and Teradata, the two largest contributors to Presto. The talk will focus on Presto’s ability to query virtually any data source via it’s connector interface. Facebook and Teradata will present some of their use cases of Presto querying various data sources, discuss the existing connectors in Presto, and describe the anatomy of a connector.

Solving Performance Problems on Hadoop

Tyler Mitchell

My presentation slides from Hadoop Summit, San Jose, June 28, 2016. See live video at http://www.makedatauseful.com/vid-solving-performance-problems-hadoop/ and follow along for context. Moving analytic workloads into production - specific technical challenges and best practices for engineering SQL in Hadoop solutions. Highlighting the next generation engineering approaches to the secret sauce we have implemented in the Actian VectorH database.

A Stock Prediction System using Open-Source Software

Fred Melo

WhereHows: Taming Metadata for 150K Datasets Over 9 Data Platforms

Mars Lan

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Spark Summit

In this presentation, we are going to talk about the state of the art infrastructure we have established at Walmart Labs for the Search product using Spark Streaming and DataFrames. First, we have been able to successfully use multiple micro batch spark streaming pipelines to update and process information like product availability, pick up today etc. along with updating our product catalog information in our search index to up to 10,000 kafka events per sec in near real-time. Earlier, all the product catalog changes in the index had a 24 hour delay, using Spark Streaming we have made it possible to see these changes in near real-time. This addition has provided a great boost to the business by giving the end-costumers instant access to features likes availability of a product, store pick up, etc. Second, we have built a scalable anomaly detection framework purely using Spark Data Frames that is being used by our data pipelines to detect abnormality in search data. Anomaly detection is an important problem not only in the search domain but also many domains such as performance monitoring, fraud detection, etc. During this, we realized that not only are Spark DataFrames able to process information faster but also are more flexible to work with. One could write hive like queries, pig like code, UDFs, UDAFs, python like code etc. all at the same place very easily and can build DataFrame template which can be used and reused by multiple teams effectively. We believe that if implemented correctly Spark Data Frames can potentially replace hive/pig in big data space and have the potential of becoming unified data language. We conclude that Spark Streaming and Data Frames are the key to processing extremely large streams of data in real-time with ease of use.

Succinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit Agarwal

Spark Summit

Delta Lake: Open Source Reliability w/ Apache Spark

George Chow

As presented: Sajith Appukuttan, Solution Architect, Databricks Sept 12, 2019 at Vancouver Spark Meetup Abstract: Delta Lake is an open-source storage layer that brings reliability to data lakes. Delta Lake offers ACID transactions, scalable metadata handling, and unifies streaming and batch data processing. It runs on top of your existing data lake and is fully compatible with Apache Spark APIs.

Building Continuously Curated Ingestion Pipelines

Arvind Prabhakar

The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...

Databricks

Many had dubbed 2020 as the decade of data. This is indeed an era of data zeitgeist. From code-centric software development 1.0, we are entering software development 2.0, a data-centric and data-driven approach, where data plays a central theme in our everyday lives. As the volume and variety of data garnered from myriad data sources continue to grow at an astronomical scale and as cloud computing offers cheap computing and data storage resources at scale, the data platforms have to match in their abilities to process, analyze, and visualize at scale and speed and with ease — this involves data paradigm shifts in processing and storing and in providing programming frameworks to developers to access and work with these data platforms. In this talk, we will survey some emerging technologies that address the challenges of data at scale, how these tools help data scientists and machine learning developers with their data tasks, why they scale, and how they facilitate the future data scientists to start quickly. In particular, we will examine in detail two open-source tools MLflow (for machine learning life cycle development) and Delta Lake (for reliable storage for structured and unstructured data). Other emerging tools such as Koalas help data scientists to do exploratory data analysis at scale in a language and framework they are familiar with as well as emerging data + AI trends in 2021. You will understand the challenges of machine learning model development at scale, why you need reliable and scalable storage, and what other open source tools are at your disposal to do data science and machine learning at scale.

Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Das...

Spark Summit

Realtime Analytical Query Processing and Predictive Model Building on High Di...

Spark Summit

Spark SQL and Mllib are optimized for running feature extraction and machine learning algorithms on row based columnar datasets through full scan but does not provide constructs for column indexing and time series analysis. For dealing with document datasets with timestamps where the features are represented as variable number of columns in each document and use-cases demand searching over columns and time to retrieve documents to generate learning models in realtime, a close integration within Spark and Lucene was needed. We introduced LuceneDAO in Spark Summit Europe 2016 to build distributed lucene shards from data frame but the time series attributes were not part of the data model. In this talk we present our extension to LuceneDAO to maintain time stamps with document-term view for search and allow time filters. Lucene shards maintain the time aware document-term view for search and vector space representation for machine learning pipelines. We used Spark as our distributed query processing engine where each query is represented as boolean combination over terms with filters on time. LuceneDAO is used to load the shards to Spark executors and power sub-second distributed document retrieval for the queries. Our synchronous API uses Spark-as-a-Service to power analytical queries while our asynchronous API uses kafka, spark streaming and HBase to power time series prediction algorithms. In this talk we will demonstrate LuceneDAO write and read performance on millions of documents with 1M+ terms and configurable time stamp aggregate columns. We will demonstrate the latency of APIs on a suite of queries generated from terms. Key takeaways from the talk will be a thorough understanding of how to make Lucene powered time aware search a first class citizen in Spark to build interactive analytical query processing and time series prediction algorithms.

Building a Federated Data Directory Platform for Public Health

Databricks

Spark in the Enterprise - 2 Years Later by Alan Saldich

Spark Summit

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

Paris Data Engineers !

MaaS (Model as a Service): Modern Streaming Data Science with Apache Metron (...

DataWorks Summit

Apache Metron (Incubating) is a streaming cybersecurity application built on Apache Storm and Hadoop. One of its core missions is to enable advanced analytics through machine learning and data science to the users. Because of the relative immaturity of data science platform infrastructure integrated into Hadoop that is oriented to streaming analytics applications, we have been forced to create the requisite platform components out of necessity, utilizing many of the pieces of the Hadoop ecosystem. In this talk, we will speak about the Metron analytics architecture and how it utilizes a custom data science model deployment and autodiscovery service that is tightly integrated with Hadoop via Yarn and Zookeeper. We will discuss how we interact with the models deployed there via a custom domain specific language that can query models as data streams past. We will generally discuss the full-stack data science tooling that has been created to enable data science at scale on an advanced analytics streaming application.

Hadoop in Validated Environment - Data Governance Initiative

DataWorks Summit

Enterprise large scale graph analytics and computing base on distribute graph...

DataWorks Summit

Graph approaches to structuring, analyzing data have been a significant area of interest, Graphs are well-suited to expressing complex interconnections and clusters of highly related entities. Large-scale graph analytics research is growing fast in recent years, to leverage Hadoop2 ecosystem for graph is a good approach, enterprise graph computer requires to store large graph and do fast computing against graph. One for the OLTP database systems which allow the user to query the graph in real-time, Hbase as the distributed NOSql database can be the backend storage to persistent large graph, the property graph stored its vertices and edges in key-value pairs in Hbase, it also provide highly reliable, scalable and fault tolerant to the data, Solr as the distributed indexing will make the query more efficient. Titan itself will handle cache, transaction; And another for the OLAP analytics systems, use TinkerPop hadoop gremlin SparkGraphComputer to processed a large graph, every vertex and edge is analyzed, a cluster-computing platform will help for the processing of large distributed in memory graph datasets. Graph DB base on Hbase/Solr and graph computing analysis base on spark is powerful for discovering valuable information about relationships in complex and large data, representing significant business opportunity in enterprise. It will help graph data analytics in a wide range of domains such as social networking, recommendation engines, advertisement optimization, knowledge representation, health care, education, and security.

Streamsets and spark at SF Hadoop User Group

Hari Shreedharan

"Who Moved my Data? - Why tracking changes and sources of data is critical to...

Cask Data

Speaker: Russ Savage, from Cask Big Data Applications Meetup, 09/14/2016 Palo Alto, CA More info here: http://www.meetup.com/BigDataApps/ Link to talk: https://youtu.be/4j78g3WvC4Y About the talk: As data lake sizes grow, and more users begin exploring and including that data in their everyday analysis, keeping track of the sources for data becomes critical. Understanding how a dataset was generated and who is using it allows users and companies to ensure their analysis is leveraging the most accurate and up to date information. In this talk, we will explore the different techniques available to keep track of your data in your data lake and demonstrate how we at Cask approached and attempted to mitigate this issue.

Spark Summit EU talk by Pat Patterson

Spark Summit

Oil and gas big data edition

Mark Kerzner

What's hot

Streamsets and spark

Hari Shreedharan

Presto: SQL-on-anything

DataWorks Summit

Solving Performance Problems on Hadoop

Tyler Mitchell

A Stock Prediction System using Open-Source Software

Fred Melo

WhereHows: Taming Metadata for 150K Datasets Over 9 Data Platforms

Mars Lan

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Spark Summit

Succinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit Agarwal

Spark Summit

Delta Lake: Open Source Reliability w/ Apache Spark

George Chow

Building Continuously Curated Ingestion Pipelines

Arvind Prabhakar

The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...

Databricks

Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Das...

Spark Summit

Realtime Analytical Query Processing and Predictive Model Building on High Di...

Spark Summit

Building a Federated Data Directory Platform for Public Health

Databricks

Spark in the Enterprise - 2 Years Later by Alan Saldich

Spark Summit

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

Paris Data Engineers !

MaaS (Model as a Service): Modern Streaming Data Science with Apache Metron (...

DataWorks Summit

Hadoop in Validated Environment - Data Governance Initiative

DataWorks Summit

Enterprise large scale graph analytics and computing base on distribute graph...

DataWorks Summit

Streamsets and spark at SF Hadoop User Group

Hari Shreedharan

"Who Moved my Data? - Why tracking changes and sources of data is critical to...

Cask Data

What's hot (20)

Streamsets and spark

Presto: SQL-on-anything

Solving Performance Problems on Hadoop

A Stock Prediction System using Open-Source Software

WhereHows: Taming Metadata for 150K Datasets Over 9 Data Platforms

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Succinct Spark: Fast Interactive Queries on Compressed RDDs by Rachit Agarwal

Delta Lake: Open Source Reliability w/ Apache Spark

Building Continuously Curated Ingestion Pipelines

The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...

Regulatory Reporting of Asset Trading Using Apache Spark-(Sudipto Shankar Das...

Realtime Analytical Query Processing and Predictive Model Building on High Di...

Building a Federated Data Directory Platform for Public Health

Spark in the Enterprise - 2 Years Later by Alan Saldich

Delta Lake OSS: Create reliable and performant Data Lake by Quentin Ambard

MaaS (Model as a Service): Modern Streaming Data Science with Apache Metron (...

Hadoop in Validated Environment - Data Governance Initiative

Enterprise large scale graph analytics and computing base on distribute graph...

Streamsets and spark at SF Hadoop User Group

"Who Moved my Data? - Why tracking changes and sources of data is critical to...

Similar to August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

Spark Summit EU talk by Pat Patterson

Spark Summit

Oil and gas big data edition

Mark Kerzner

The Maturity Model: Taking the Growing Pains Out of Hadoop

Inside Analysis

The Briefing Room with Rick van der Lans and Think Big, a Teradata Company Live Webcast on June 16, 2015 Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=197f8106531874cc5c14081ca214eaff Hadoop is arguably one of the most disruptive technologies of the last decade. Once lauded solely for its ability to transform the speed of batch processing, it has marched steadily forward and promulgated an array of performance-enhancing accessories, notably Spark and YARN. Hadoop has evolved into much more than a file system and batch processor, and it now promises to stand as the data management and analytics backbone for enterprises. Register for this episode of The Briefing Room to learn from veteran Analyst Rick van der Lans, as he discusses the emerging roles of Hadoop within the analytics ecosystem. He’ll be briefed by Ron Bodkin of Think Big, a Teradata Company, who will explore Hadoop’s maturity spectrum, from typical entry use cases all the way up the value chain. He’ll show how enterprises that already use Hadoop in production are finding new ways to exploit its power and build creative, dynamic analytics environments. Visit InsideAnalysis.com for more information.

BDW Chicago 2016 - Ramu Kalvakuntla, Sr. Principal - Technical - Big Data Pra...

Big Data Week

We all are aware of the challenges enterprises are having with growing data and silo’d data stores. Business is not able to make reliable decisions with un-trusted data and on top of that, they don’t have access to all data within and outside their enterprise to stay ahead of the competition and make key decisions in their business This session will take a deep dive into current challenges business are having today and how to build a Modern Data Architecture using emerging technologies such as Hadoop, Spark, NoSQL data stores, MPP Data stores and scalable and cost effective cloud solutions such as AWS, Azure and Bigstep.

Balancing data democratization with comprehensive information governance: bui...

DataWorks Summit

If information is the new oil, then governance is its “safety data sheet.” As demand for data as the raw material for competitive differentiation continues to rise in enterprises, enterprises are having bigger challenges identifying and valuing data and ensuring its appropriate use to extract the right information. In order for organizations to make effective business decisions, organizations need to have trust in their data so that they can impute the right value and use it for the right purposes while satisfying any organizational or regulatory mandates. A number of analytics and data science initiatives fail to reach their potential due to lack of an information governance framework in place. Robust information governance capabilities can help organizations develop trust in their data and empower them to make decisions confidently. In this session Sanjeev Mohan, Research Analyst at Gartner, and Srikanth Venkat, Sr. Director of Product Management at Hortonworks, will walk you through an end-to-end architectural blueprint for information governance and best practices for helping organizations understand, secure, and govern diverse types of data in enterprise data lakes. Speaker Sanjeev Mohan, Gartner, Research Analyst Srikanth Venkat, Hortonworks, Senior Director, Product Management In this session Sanjeev Mohan, Research Analyst at Gartner, and Srikanth Venkat, Sr. Director of Product Management at Hortonworks, will walk you through an end-to-end architectural blueprint for information governance and best practices for helping organizations understand, secure, and govern diverse types of data in enterprise data lakes.

Hadoop 2.0: YARN to Further Optimize Data Processing

Hortonworks

Data is exponentially increasing in both types and volumes, creating opportunities for businesses. Watch this video and learn from three Big Data experts: John Kreisa, VP Strategic Marketing at Hortonworks, Imad Birouty, Director of Technical Product Marketing at Teradata and John Haddad, Senior Director of Product Marketing at Informatica. Multiple systems are needed to exploit the variety and volume of data sources, including a flexible data repository. Learn more about: - Apache Hadoop 2 and YARN - Data Lakes - Intelligent data management layers needed to manage metadata and usage patterns as well as track consumption across these data platforms.

Hortonworks and HP Vertica Webinar

Hortonworks

Setting Up the Data Lake

Caserta

Big data journey to the cloud maz chaudhri 5.30.18

Cloudera, Inc.

8.17.11 big data and hadoop with informatica slideshare

Julianna DeLua

2015 02 12 talend hortonworks webinar challenges to hadoop adoption

Hortonworks

Hadoop is no longer optional. Companies of all sizes are in various phases of their own Big Data journey. Whether you are just starting to explore the platform or have multiple clusters up and running, everyone is presented with a similar challenge - developing their internal skillset. Hadoop specialists are hard to find. Hand coding is too prone to error when it comes to storing, integrating or analyzing your data. However, it doesn’t need to be this difficult. In this recorded webinar, Talend and Hortonworks help you learn how to unify all your data in Hadoop, with no specialized Big Data skills. Find the recording here. www.talend.com/resources/webinars/challenges-to-hadoop-adoption-if-you-can-dream-it-you-can-build-it This webinar covers: How Hadoop opens a new world of analytic applications, How to bridge the skills gap with our Big Data solutions, Experience a real-world, simple technical demo

Rev_3 Components of a Data WarehouseRyan Andhavarapu

CWIN17 India / Bigdata architecture yashowardhan sowale

Capgemini

A Winning Strategy for the Digital Economy

Eric Kavanagh

The speed of innovation today creates tremendous opportunities for some, existential threats for others. Companies that win create their own success by leveraging modern data platforms. While architectures vary, the foundation is often in-memory, and the latency is real-time. Register for this Special Edition of The Briefing Room to hear veteran Analyst Dr. Robin Bloor explain how today's data platforms enable the modern enterprise in groundbreaking ways. He'll be briefed by Chris Hallenbeck of SAP who will demonstrate how forward-looking companies are leveraging real-time data platforms to achieve operational excellence, make decisions faster, and find new ways to innovate.

Data lake benefits

Ricky Barron

Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...

Hortonworks

Many enterprises are turning to Apache Hadoop to enable Big Data Analytics and reduce the costs of traditional data warehousing. Yet, it is hard to succeed when 80% of the time is spent on moving data and only 20% on using it. It’s time to swap the 80/20! The Big Data experts at Attunity and Hortonworks have a solution for accelerating data movement into and out of Hadoop that enables faster time-to-value for Big Data projects and a more complete and trusted view of your business. Join us to learn how this solution can work for you.

IoT Crash Course Hadoop Summit SJ

Daniel Madrigal

Solving Big Data Problems using Hortonworks

DataWorks Summit/Hadoop Summit

Predictive Analytics - Big Data Warehousing Meetup

Caserta

Predictive analytics has always been about the future, and the age of big data has made that future an increasingly dynamic place, filled with opportunity and risk. The evolution of advanced analytics technologies and the continual development of new analytical methodologies can help to optimize financial results, enable systems and services based on machine learning, obviate or mitigate fraud and reduce cybersecurity risks, among many other things. Caserta Concepts, Zementis, and guest speaker from FICO presented the strategies, technologies and use cases driving predictive analytics in a big data environment. For more information, visit www.casertaconcepts.com or contact us at info@casertaconcepts.com

Keeping the Pulse of Your Data: Why You Need Data Observability

Precisely

With the explosive growth of DataOps to drive faster and better-informed business decisions, proactively understanding the health of your data is more important than ever. Data observability is one of the foundational capabilities of DataOps and an emerging discipline used to expose anomalies in data by continuously monitoring and testing data using artificial intelligence and machine learning to trigger alerts when issues are discovered. Join Paul Rasmussen and Shalaish Koul from Precisely, to learn how data observability can be used as part of a DataOps strategy to prevent data issues from wreaking havoc on your analytics and ensure that your organization can confidently rely on the data used for advanced analytics and business intelligence. Topics you will hear addressed in this webinar: Data observability – what is it and how it is different from other monitoring solutions Why now is the time to incorporate data observability into your DataOps strategy How data observability helps prevent data issues from impacting downstream analytics Examples of how data observability can be used to prevent real-world issues

Similar to August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector (20)

Spark Summit EU talk by Pat Patterson

Oil and gas big data edition

The Maturity Model: Taking the Growing Pains Out of Hadoop

BDW Chicago 2016 - Ramu Kalvakuntla, Sr. Principal - Technical - Big Data Pra...

Balancing data democratization with comprehensive information governance: bui...

Hadoop 2.0: YARN to Further Optimize Data Processing

Hortonworks and HP Vertica Webinar

Setting Up the Data Lake

Big data journey to the cloud maz chaudhri 5.30.18

8.17.11 big data and hadoop with informatica slideshare

2015 02 12 talend hortonworks webinar challenges to hadoop adoption

Rev_3 Components of a Data Warehouse

CWIN17 India / Bigdata architecture yashowardhan sowale

A Winning Strategy for the Digital Economy

Data lake benefits

Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...

IoT Crash Course Hadoop Summit SJ

Solving Big Data Problems using Hortonworks

Predictive Analytics - Big Data Warehousing Meetup

Keeping the Pulse of Your Data: Why You Need Data Observability

More from Yahoo Developer Network

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

Similar to August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector (20)

More from Yahoo Developer Network

More from Yahoo Developer Network (20)

Recently uploaded

Recently uploaded (20)

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector