Introduction to Microsoft HDInsight and BI Tools

•

2 likes•2,569 views

This document discusses Hortonworks Data Platform (HDP) for Windows. It includes an agenda for the presentation which covers an introduction to HDP for Windows, integrating HDP with Microsoft tools, and a demo. The document lists the speakers and provides information on Windows support for Hadoop components. It describes what is included in HDP for Windows, such as deployment choices and full interoperability across platforms. Integration with Microsoft tools like SQL Server, Excel, and Power BI is highlighted. A demo of using Excel to interact with HDP is promised.

Technology

HDP for Windows

Hadoop Summit
Amsterdam

© Hortonworks Inc. 2013

Our Speakers

Abhijit Lele
Solution Engineer

Rohit Bakhshi
Product Manager

Page 2
© Hortonworks Inc. 2013

Agenda

• HDP for Windows

• Integrating with the Microsoft Tool Ecosystem

• Demo

• Q&A

Page 3
© Hortonworks Inc. 2013

Introducing HDP for Windows

OPERATIONAL DATA Hortonworks
SERVICES SERVICES
Data Platform (HDP)
Manage &
Manage & Store,
Store,
Operate at
Operate at Process and
Process and For Windows
Scale
Scale Access Data
Access Data

• 100% Open Source
HADOOP CORE
Distributed Enterprise Hadoop
Storage & Processing

• Component and version
PLATFORM SERVICES Enterprise Readiness
compatible with Microsoft
HDInsight
HORTONWORKS
DATA PLATFORM (HDP) • Availability
For Windows
• Beta release available now

• GA early 2Q 2013

Page 4
© Hortonworks Inc. 2013

Windows support in Apache

Component Version

Hadoop 1.0.3

Pig 0.9.3

Hive 0.9.0

HCatalog 0.4.1

WebHCat 0.1.4

Sqoop 1.4.2

Oozie 3.2.0

Initial Hadoop Core Apache JIRA:
https://issues.apache.org/jira/browse/HADOOP-8079

Apache Hadoop branch-1-win github:
https://github.com/apache/hadoop-common/tree/branch-1-win

Apache Hadoop branch-trunk-win github:
https://github.com/apache/hadoop-common/tree/branch-trunk-win

© Hortonworks Inc. 2013

What do you get?
• More deployment choices
• Hadoop for Windows is for on-premise deployment
– Good fit for organizations with Hadoop operational experience
– Next step for those who are ready to move from POC to production
• Use HDInsight for public and private cloud
deployments
– HDInsight Service -> Windows Azure– available in Preview today
– HDInsight Server -> for interoperability across platforms of Hadoop, with
Microsoft tools, on premise – Developer Preview available today
• Full interoperability across platforms
• Created through partnership between Hortonworks
and Microsoft
– Eighteen months of development time

Page 6
© Hortonworks Inc. 2013

Cluster Management
Start/Stop via Services Administration Tool

© Hortonworks Inc. 2013

Functionality Consistency Across Platforms

© Hortonworks Inc. 2013

Agenda

• HDP for Windows

• Integrating with the Microsoft Tool Ecosystem

• Demo

• Q&A

Page 9
© Hortonworks Inc. 2013

Seamless Interoperability with Your Microsoft Tools

• Integrated with Microsoft
APPLICATIONS

tools for native big data
analysis
Microsoft Applications – Bi-directional connectors for SQL Server
and SQL Azure through SQOOP
– Excel ODBC integration through Hive

• Addressing demand for
DATA SYSTEMS

HORTONWORKS
DATA PLATFORM
Hadoop on Windows
For Windows – Ideal for Windows customers with
Hadoop operational experience

• Enables all common Hadoop
workloads
– Data refinement and ETL offload for
DATA SOURCES

high-volume data landing
Traditional Sources New Sources
(web logs, email, sensor data, social media)
– Data exploration for discovery of new
OLTP, (RDBMS, OLTP, OLAP) MOBILE
POS DATA business opportunities
SYSTEMS

Page 10
© Hortonworks Inc. 2013

Hadoop and Microsoft Connectivity

•Infrastructure Components
• Active Directory
• System Center
•Microsoft Data Connectivity
• Sqoop Connector
• SQL Server / SQL Parallel Data Warehouse
• Azure Storage / Azure Data Market
•Microsoft Business Intelligence (BI)
• ODBC Connectivity
© Hortonworks Inc. 2013

Excel Hive-Add on

© Hortonworks Inc. 2013

Hadoop/Hive and Analysis Services

•Delivers cost-efficient OLAP and data
mining functionality
•Supports ad-hoc data support queries in
Microsoft Excel and dashboards
•Offers exceptional query performance over
vast amounts of data
•SSAS to HIVE paper
–http://msdn.microsoft.com/en-
us/library/jj710329.aspx

© Hortonworks Inc. 2013

Interoperability: Looking Ahead, PolyBase

• Single query for relational &
Hadoop data

• Process data in place

• Seamless: Regular T-SQL PolyBase
command

• Future expansion to other data
sources

© Hortonworks Inc. 2013

Agenda

• HDP for Windows

• Integrating with the Microsoft Tool Ecosystem

• Demo

• Q&A

Page 17
© Hortonworks Inc. 2013

Demo Time!

Excel integration with HDP
• Interact with HDP through Excel
• Use Data Explorer to explore and turn raw data
into valuable information

Page 18
© Hortonworks Inc. 2013

According to IDC, Windows Servers run more than 50% of the servers in the Enterprise Data Center. Hortonworks has worked closely with Microsoft to port Apache Hadoop to Windows to enable organizations to take advantage of this emerging Big Data technology. Join us in this informative webinar to hear about the new Hortonworks Data Platform for Windows. In less than an hour, you’ll learn: -Key capabilities available in Hortonworks Data Platform for Windows -How HDP for Windows integrates with Microsoft tools -Key workloads and use cases for driving Hadoop today

Hadoop Reporting and Analysis - Jaspersoft

Hortonworks

Introduction to Microsoft Azure HD Insight by Dattatrey Sindhol

HARMAN Services

Apache Hadoop on the Open Cloud

Hortonworks

C* Summit EU 2013: Leveraging the Power of Cassandra: Operational Reporting a...

DataStax Academy

Data Lake for the Cloud: Extending your Hadoop Implementation

Hortonworks

As more applications are created using Apache Hadoop that derive value from the new types of data from sensors/machines, server logs, click-streams, and other sources, the enterprise "Data Lake" forms with Hadoop acting as a shared service. While these Data Lakes are important, a broader life-cycle needs to be considered that spans development, test, production, and archival and that is deployed across a hybrid cloud architecture. If you have already deployed Hadoop on-premise, this session will also provide an overview of the key scenarios and benefits of joining your on-premise Hadoop implementation with the cloud, by doing backup/archive, dev/test or bursting. Learn how you can get the benefits of an on-premise Hadoop that can seamlessly scale with the power of the cloud.

Data Discovery, Visualization, and Apache Hadoop

Hortonworks

Microsoft and Hortonworks Delivers the Modern Data Architecture for Big Data

Hortonworks

Hadoop is a great platform for storing and processing massive amounts of data. Elasticsearch is the ideal solution for Searching and Visualizing the same data. Join us to learn how you can leverage the full power of both platforms to maximize the value of your Big Data. In this webinar we'll walk you through: How Elasticsearch fits in the Modern Data Architecture. A demo of Elasticsearch and Hortonworks Data Platform. Best practices for combining Elasticsearch and Hortonworks Data Platform to extract maximum insights from your data.

Building a Big Data platform with the Hadoop ecosystem

Gregg Barrett

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

Innovative Management Services

Mr. Slim Baltagi is a Systems Architect at Hortonworks, with over 4 years of Hadoop experience working on 9 Big Data projects: Advanced Customer Analytics, Supply Chain Analytics, Medical Coverage Discovery, Payment Plan Recommender, Research Driven Call List for Sales, Prime Reporting Platform, Customer Hub, Telematics, Historical Data Platform; with Fortune 100 clients and global companies from Financial Services, Insurance, Healthcare and Retail. Mr. Slim Baltagi has worked in various architecture, design, development and consulting roles at. Accenture, CME Group, TransUnion, Syntel, Allstate, TransAmerica, Credit Suisse, Chicago Board Options Exchange, Federal Reserve Bank of Chicago, CNA, Sears, USG, ACNielsen, Deutshe Bahn. Mr. Baltagi has also over 14 years of IT experience with an emphasis on full life cycle development of Enterprise Web applications using Java and Open-Source software. He holds a master’s degree in mathematics and is an ABD in computer science from Université Laval, Québec, Canada. Languages: Java, Python, JRuby, JEE , PHP, SQL, HTML, XML, XSLT, XQuery, JavaScript, UML, JSON Databases: Oracle, MS SQL Server, MYSQL, PostreSQL Software: Eclipse, IBM RAD, JUnit, JMeter, YourKit, PVCS, CVS, UltraEdit, Toad, ClearCase, Maven, iText, Visio, Japser Reports, Alfresco, Yslow, Terracotta, Toad, SoapUI, Dozer, Sonar, Git Frameworks: Spring, Struts, AppFuse, SiteMesh, Tiles, Hibernate, Axis, Selenium RC, DWR Ajax , Xstream Distributed Computing/Big Data: Hadoop, MapReduce, HDFS, Hive, Pig, Sqoop, HBase, R, RHadoop, Cloudera CDH4, MapR M7, Hortonworks HDP 2.1

YARN Ready: Integrating to YARN with Tez

Hortonworks

Oncrawl elasticsearch meetup france #12

Tanguy MOAL

Presentation detailing how Elasticsearch is involved in Oncrawl, a SaaS solution for easy SEO monitoring. The presentation explains how the application is built, and how it integrates Elasticsearch, a powerful general purpose search engine. Oncrawl is data centric and elasticsearch is used as an analytics engine rather than a full text search engine. The application uses Apache Hadoop and Apache Nutch for the crawl pipeline and data analysis. Oncrawl is a Cogniteev solution.

Hdp r-google charttools-webinar-3-5-2013 (2)Hortonworks

Hadoop as Data Refinery - Steve Loughran

JAX London

Apache Hadoop is often described as a "Big Data Platform" but what does that mean? One way to better understand Hadoop is to talk about how Hadoop is used. This talk discusses using Hadoop as a "Data Refinery", which is a common use case. The concept is very much like a traditional oil refinery except with data, pulling in large quantities of "crude data" over pipelines, refining some into useful business intelligence; refining other pieces into slightly less crude data that stays in the cluster until needed later. This metaphor proves useful when considering how Hadoop could be adopted in an organisation that already has data warehousing and business intelligence systems -and when contemplating how to hook up a Hadoop cluster to the sources of data inside and outside that organisation. A key point to remember is that storing data in Hadoop is not a means to an end any more than storing data in a database is: it is extracting information from that data. Using Hadoop as a front end "data refinery" means that it can integrate with existing Business Intelligence systems, while providing the platform for new applications.

Hadoop as data refinery

Steve Loughran

What is hadoop

Asis Mohanty

Rescue your Big Data from Downtime with HP Operations Bridge and Apache Hadoop

Hortonworks

Hortonworks and HP Vertica Webinar

Hortonworks

Introduction to Hortonworks Data PlatformHortonworks

The Modern Data Architecture for Advanced Business Intelligence with Hortonwo...

Hortonworks

There certainly is no shortage of hype when it comes to the term “Big Data”. One thing we can be sure of is that massive data volumes are driving a new modern data architecture that includes Hadoop in the mix. But what does that architecture look like for Business Intelligence Data Strategy? Join Hortonworks and MicroStrategy, where we’ll: • Discuss the modern architecture for Business Intelligence on top of Hadoop as a data source. • Learn how our joint solution helps enterprises store, process and analyze vast amounts of structured and unstructured data to deliver business insights throughout an organization. • Discover what new benefits Hadoop 2.0 offers and how the MicroStrategy Analytics platform leverages those new features to improve performance, achieve faster access times, and allow for true interactive visual data discovery.

Software Architecture and Predictive Models in R

Harlan Harris

Bigger Data For Your Budget

Hortonworks

Hadoop Powers Modern Enterprise Data ArchitecturesDataWorks Summit

Hortonworks and Red Hat Webinar - Part 2

Hortonworks

Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (...

Hortonworks

Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...

Hortonworks

Many enterprises are turning to Apache Hadoop to enable Big Data Analytics and reduce the costs of traditional data warehousing. Yet, it is hard to succeed when 80% of the time is spent on moving data and only 20% on using it. It’s time to swap the 80/20! The Big Data experts at Attunity and Hortonworks have a solution for accelerating data movement into and out of Hadoop that enables faster time-to-value for Big Data projects and a more complete and trusted view of your business. Join us to learn how this solution can work for you.

Introduction to Designing and Building Big Data Applications

Cloudera, Inc.

Apache Hadoop Now Next and Beyond

DataWorks Summit

With the rise of Apache Hadoop, a next-generation enterprise data architecture is emerging that connects the systems powering business transactions and business intelligence. Hadoop is uniquely capable of storing, aggregating, and refining multi-structured data sources into formats that fuel new business insights. Apache Hadoop is fast becoming the defacto platform for processing Big Data. Hadoop started from a relatively humble beginning as a point solution for small search systems. Its growth into an important technology to the broader enterprise community dates back to Yahoo’s 2006 decision to evolve Hadoop into a system for solving its internet scale big data problems. Eric will discuss the current state of Hadoop and what is coming from a development standpoint as Hadoop evolves to meet more workloads.

Big Data Analytics - Is Your Elephant Enterprise Ready?

Hortonworks

Hadoop’s cost effective scalability and flexibility to analyze all data types is driving organizations everywhere to embrace big data analytics. From proof of concept to deployment across the enterprise, join Datameer and Hortonworks as we answer the ‘now what?’ when rolling out your Hadoop big data analytics project. This webinar will address critical project components such as data security, data privacy, high availability, user training and use case development.

What's hot

Combine Apache Hadoop and Elasticsearch to Get the Most of Your Big Data

Hortonworks

Building a Big Data platform with the Hadoop ecosystem

Gregg Barrett

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

Innovative Management Services

YARN Ready: Integrating to YARN with Tez

Hortonworks

Oncrawl elasticsearch meetup france #12

Tanguy MOAL

Hdp r-google charttools-webinar-3-5-2013 (2)Hortonworks

Hadoop as Data Refinery - Steve Loughran

JAX London

Hadoop as data refinery

Steve Loughran

What is hadoop

Asis Mohanty

Rescue your Big Data from Downtime with HP Operations Bridge and Apache Hadoop

Hortonworks

Hortonworks and HP Vertica Webinar

Hortonworks

Introduction to Hortonworks Data PlatformHortonworks

The Modern Data Architecture for Advanced Business Intelligence with Hortonwo...

Hortonworks

Software Architecture and Predictive Models in R

Harlan Harris

Bigger Data For Your Budget

Hortonworks

Hadoop Powers Modern Enterprise Data ArchitecturesDataWorks Summit

Hortonworks and Red Hat Webinar - Part 2

Hortonworks

Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (...

Hortonworks

Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...

Hortonworks

Introduction to Designing and Building Big Data Applications

Cloudera, Inc.

What's hot (20)

Combine Apache Hadoop and Elasticsearch to Get the Most of Your Big Data

Building a Big Data platform with the Hadoop ecosystem

Open-BDA Hadoop Summit 2014 - Mr. Slim Baltagi (Building a Modern Data Archit...

YARN Ready: Integrating to YARN with Tez

Oncrawl elasticsearch meetup france #12

Hdp r-google charttools-webinar-3-5-2013 (2)

Hadoop as Data Refinery - Steve Loughran

Hadoop as data refinery

What is hadoop

Rescue your Big Data from Downtime with HP Operations Bridge and Apache Hadoop

Hortonworks and HP Vertica Webinar

Introduction to Hortonworks Data Platform

The Modern Data Architecture for Advanced Business Intelligence with Hortonwo...

Software Architecture and Predictive Models in R

Bigger Data For Your Budget

Hadoop Powers Modern Enterprise Data Architectures

Hortonworks and Red Hat Webinar - Part 2

Discover hdp 2.2: Data storage innovations in Hadoop Distributed Filesystem (...

Webinar - Accelerating Hadoop Success with Rapid Data Integration for the Mod...

Introduction to Designing and Building Big Data Applications

Similar to Introduction to Microsoft HDInsight and BI Tools

Apache Hadoop Now Next and Beyond

DataWorks Summit

Big Data Analytics - Is Your Elephant Enterprise Ready?

Hortonworks

OSDC 2013 | Introduction into Hadoop by Olivier Renault

NETWAYS

How do I deploy, manage and monitor Apache Hadoop within my Data Center? Olivier Renault will help you kick start your deployment by sharing the experience that the Hortonworks team have from deployment in the Yahoo Data Center and many other enterprise production deployments. During this talk, we will present Ambari the Open Source management solution by Apache which make deploying and monitoring Hadoop extremely easy.

Hadoop Demystified + MapReduce (Java and C#), Pig, and Hive Demos

Lester Martin

YARN - Strata 2014

Hortonworks

Part of the core Hadoop project, YARN is the architectural center of Hadoop that allows multiple data processing engines such as interactive SQL, real-time streaming, data science and batch processing to handle data stored in a single platform, unlocking an entirely new approach to analytics. It is the foundation of the new generation of Hadoop and is enabling organizations everywhere to realize a Modern Data Architecture.

Hortonworks - What's Possible with a Modern Data Architecture?

Hortonworks

Non-Stop Hadoop for Hortonworks

Hortonworks

Mrinal devadas, Hortonworks Making Sense Of Big DataPatrickCrompton

Supporting Financial Services with a More Flexible Approach to Big Data

Hortonworks

Financial services companies can reap tremendous benefits from 'Big Data' and they have moved quickly to deploy it. But these companies also place heavy demands on 'Big Data' infrastructure for flexibility, reliability and performance. In this webinar, Hortonworks joins WANDisco to look at three examples of using 'Big Data' to get a more comprehensive view of customer behavior and activity in the banking and insurance industries. Then we'll pull out the common threads from these examples, and see how a flexible next-generation Hadoop architecture lets you get a step up on improving your business performance. Join us to learn: How to leverage data from across an entire global enterprise How to analyze a wide variety of structured and unstructured data to get quick, meaningful answers to critical questions What industry leaders have put in place

Transform Your Business with Big Data and Hortonworks

Pactera_US

Why hadoop for data science?Hortonworks

Transform You Business with Big Data and HortonworksHortonworks

Introduction to the Hortonworks YARN Ready Program

Hortonworks

Create a Smarter Data Lake with HP Haven and Apache Hadoop

Hortonworks

An organization’s information is spread across multiple repositories, on-premise and in the cloud, with limited ability to correlate information and derive insights. The Smart Content Hub solution from HP and Hortonworks enables a shared content infrastructure that transparently synchronizes information with existing systems and offers an open standards-based platform for deep analysis and data monetization. - Leverage 100% of your data: Text, images, audio, video, and many more data types can be automatically consumed and enriched using HP Haven (powered by HP IDOL and HP Vertica), making it possible to integrate this valuable content and insights into various line of business applications. - Democratize and enable multi-dimensional content analysis: - Empower your analysts, business users, and data scientists to search and analyze Hadoop data with ease, using the 100% open source Hortonworks Data Platform. - Extend the enterprise data warehouse: Synchronize and manage content from content management systems, and crack open the files in whatever format they happen to be in. - Dramatically reduce complexity with enterprise-ready SQL engine: Tap into the richest analytics that support JOINs, complex data types, and other capabilities only available with HP Vertica SQL on the Hortonworks Data Platform. Speakers: - Ajay Singh, Director, Technical Channels, Hortonworks - Will Gardella, Product Management, HP Big Data

201305 hadoop jpl-v3Eric Baldeschwieler

Coordinating the Many Tools of Big Data - Apache HCatalog, Apache Pig and Apa...

Big Data Spain

Hadoop: today and tomorrow

Steve Loughran

Cloud Austin Meetup - Hadoop like a champion

Ameet Paranjape

Strata feb2013

alanfgates

Building a Modern Data Architecture with Enterprise Hadoop

Slim Baltagi

Similar to Introduction to Microsoft HDInsight and BI Tools (20)

Apache Hadoop Now Next and Beyond

Big Data Analytics - Is Your Elephant Enterprise Ready?

OSDC 2013 | Introduction into Hadoop by Olivier Renault

Hadoop Demystified + MapReduce (Java and C#), Pig, and Hive Demos

YARN - Strata 2014

Hortonworks - What's Possible with a Modern Data Architecture?

Non-Stop Hadoop for Hortonworks

Mrinal devadas, Hortonworks Making Sense Of Big Data

Supporting Financial Services with a More Flexible Approach to Big Data

Transform Your Business with Big Data and Hortonworks

Why hadoop for data science?

Transform You Business with Big Data and Hortonworks

Introduction to the Hortonworks YARN Ready Program

Create a Smarter Data Lake with HP Haven and Apache Hadoop

201305 hadoop jpl-v3

Coordinating the Many Tools of Big Data - Apache HCatalog, Apache Pig and Apa...

Hadoop: today and tomorrow

Cloud Austin Meetup - Hadoop like a champion

Strata feb2013

Building a Modern Data Architecture with Enterprise Hadoop

More from DataWorks Summit

Data Science Crash Course

DataWorks Summit

Introduction: This workshop will provide a hands-on introduction to Machine Learning (ML) with an overview of Deep Learning (DL). Format: An introductory lecture on several supervised and unsupervised ML techniques followed by light introduction to DL and short discussion what is current state-of-the-art. Several python code samples using the scikit-learn library will be introduced that users will be able to run in the Cloudera Data Science Workbench (CDSW). Objective: To provide a quick and short hands-on introduction to ML with python’s scikit-learn library. The environment in CDSW is interactive and the step-by-step guide will walk you through setting up your environment, to exploring datasets, training and evaluating models on popular datasets. By the end of the crash course, attendees will have a high-level understanding of popular ML algorithms and the current state of DL, what problems they can solve, and walk away with basic hands-on experience training and evaluating ML models. Prerequisites: For the hands-on portion, registrants must bring a laptop with a Chrome or Firefox web browser. These labs will be done in the cloud, no installation needed. Everyone will be able to register and start using CDSW after the introductory lecture concludes (about 1hr in). Basic knowledge of python highly recommended.

Floating on a RAFT: HBase Durability with Apache Ratis

DataWorks Summit

In a world with a myriad of distributed storage systems to choose from, the majority of Apache HBase clusters still rely on Apache HDFS. Theoretically, any distributed file system could be used by HBase. One major reason HDFS is predominantly used are the specific durability requirements of HBase's write-ahead log (WAL) and HDFS providing that guarantee correctly. However, HBase's use of HDFS for WALs can be replaced with sufficient effort. This talk will cover the design of a "Log Service" which can be embedded inside of HBase that provides a sufficient level of durability that HBase requires for WALs. Apache Ratis (incubating) is a library-implementation of the RAFT consensus protocol in Java and is used to build this Log Service. We will cover the design choices of the Ratis Log Service, comparing and contrasting it to other log-based systems that exist today. Next, we'll cover how the Log Service "fits" into HBase and the necessary changes to HBase which enable this. Finally, we'll discuss how the Log Service can simplify the operational burden of HBase.

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

DataWorks Summit

Utilizing Apache NiFi we read various open data REST APIs and camera feeds to ingest crime and related data real-time streaming it into HBase and Phoenix tables. HBase makes an excellent storage option for our real-time time series data sources. We can immediately query our data utilizing Apache Zeppelin against Phoenix tables as well as Hive external tables to HBase. Apache Phoenix tables also make a great option since we can easily put microservices on top of them for application usage. I have an example Spring Boot application that reads from our Philadelphia crime table for front-end web applications as well as RESTful APIs. Apache NiFi makes it easy to push records with schemas to HBase and insert into Phoenix SQL tables. Resources: https://community.hortonworks.com/articles/54947/reading-opendata-json-and-storing-into-phoenix-tab.html https://community.hortonworks.com/articles/56642/creating-a-spring-boot-java-8-microservice-to-read.html https://community.hortonworks.com/articles/64122/incrementally-streaming-rdbms-data-to-your-hadoop.html

HBase Tales From the Trenches - Short stories about most common HBase operati...

DataWorks Summit

Whilst HBase is the most logical answer for use cases requiring random, realtime read/write access to Big Data, it may not be so trivial to design applications that make most of its use, neither the most simple to operate. As it depends/integrates with other components from Hadoop ecosystem (Zookeeper, HDFS, Spark, Hive, etc) or external systems ( Kerberos, LDAP), and its distributed nature requires a "Swiss clockwork" infrastructure, many variables are to be considered when observing anomalies or even outages. Adding to the equation there's also the fact that HBase is still an evolving product, with different release versions being used currently, some of those can carry genuine software bugs. On this presentation, we'll go through the most common HBase issues faced by different organisations, describing identified cause and resolution action over my last 5 years supporting HBase to our heterogeneous customer base.

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

DataWorks Summit

LocationTech GeoMesa enables spatial and spatiotemporal indexing and queries for HBase and Accumulo. In this talk, after an overview of GeoMesa’s capabilities in the Cloudera ecosystem, we will dive into how GeoMesa leverages Accumulo’s Iterator interface and HBase’s Filter and Coprocessor interfaces. The goal will be to discuss both what spatial operations can be pushed down into the distributed database and also how the GeoMesa codebase is organized to allow for consistent use across the two database systems.

Managing the Dewey Decimal System

DataWorks Summit

Practical NoSQL: Accumulo's dirlist Example

DataWorks Summit

Many individuals/organizations have a desire to utilize NoSQL technology, but often lack an understanding of how the underlying functional bits can be utilized to enable their use case. This situation can result in drastic increases in the desire to put the SQL back in NoSQL. Since the initial commit, Apache Accumulo has provided a number of examples to help jumpstart comprehension of how some of these bits function as well as potentially help tease out an understanding of how they might be applied to a NoSQL friendly use case. One very relatable example demonstrates how Accumulo could be used to emulate a filesystem (dirlist). In this session we will walk through the dirlist implementation. Attendees should come away with an understanding of the supporting table designs, a simple text search supporting a single wildcard (on file/directory names), and how the dirlist elements work together to accomplish its feature set. Attendees should (hopefully) also come away with a justification for sometimes keeping the SQL out of NoSQL.

HBase Global Indexing to support large-scale data ingestion at Uber

DataWorks Summit

Data serves as the platform for decision-making at Uber. To facilitate data driven decisions, many datasets at Uber are ingested in a Hadoop Data Lake and exposed to querying via Hive. Analytical queries joining various datasets are run to better understand business data at Uber. Data ingestion, at its most basic form, is about organizing data to balance efficient reading and writing of newer data. Data organization for efficient reading involves factoring in query patterns to partition data to ensure read amplification is low. Data organization for efficient writing involves factoring the nature of input data - whether it is append only or updatable. At Uber we ingest terabytes of many critical tables such as trips that are updatable. These tables are fundamental part of Uber's data-driven solutions, and act as the source-of-truth for all the analytical use-cases across the entire company. Datasets such as trips constantly receive updates to the data apart from inserts. To ingest such datasets we need a critical component that is responsible for bookkeeping information of the data layout, and annotates each incoming change with the location in HDFS where this data should be written. This component is called as Global Indexing. Without this component, all records get treated as inserts and get re-written to HDFS instead of being updated. This leads to duplication of data, breaking data correctness and user queries. This component is key to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. This component will need to have strong consistency and provide large throughputs for index writes and reads. At Uber, we have chosen HBase to be the backing store for the Global Indexing component and is a critical component in allowing us to scaling our jobs where we are now handling greater than 500 billion writes a day in our current ingestion systems. In this talk, we will discuss data@Uber and expound more on why we built the global index using Apache Hbase and how this helps to scale out our cluster usage. We’ll give details on why we chose HBase over other storage systems, how and why we came up with a creative solution to automatically load Hfiles directly to the backend circumventing the normal write path when bootstrapping our ingestion tables to avoid QPS constraints, as well as other learnings we had bringing this system up in production at the scale of data that Uber encounters daily.

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

DataWorks Summit

Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions. These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

DataWorks Summit

Cybersecurity requires an organization to collect data, analyze it, and alert on cyber anomalies in near real-time. This is a challenging endeavor when considering the variety of data sources which need to be collected and analyzed. Everything from application logs, network events, authentications systems, IOT devices, business events, cloud service logs, and more need to be taken into consideration. In addition, multiple data formats need to be transformed and conformed to be understood by both humans and ML/AI algorithms. To solve this problem, the Aetna Global Security team developed the Unified Data Platform based on Apache NiFi, which allows them to remain agile and adapt to new security threats and the onboarding of new technologies in the Aetna environment. The platform currently has over 60 different data flows with 95% doing real-time ETL and handles over 20 billion events per day. In this session learn from Aetna’s experience building an edge to AI high-speed data pipeline with Apache NiFi.

Supporting Apache HBase : Troubleshooting and Supportability Improvements

DataWorks Summit

Security Framework for Multitenant Architecture

DataWorks Summit

In the healthcare sector, data security, governance, and quality are crucial for maintaining patient privacy and ensuring the highest standards of care. At Florida Blue, the leading health insurer of Florida serving over five million members, there is a multifaceted network of care providers, business users, sales agents, and other divisions relying on the same datasets to derive critical information for multiple applications across the enterprise. However, maintaining consistent data governance and security for protected health information and other extended data attributes has always been a complex challenge that did not easily accommodate the wide range of needs for Florida Blue’s many business units. Using Apache Ranger, we developed a federated Identity & Access Management (IAM) approach that allows each tenant to have their own IAM mechanism. All user groups and roles are propagated across the federation in order to determine users’ data entitlement and access authorization; this applies to all stages of the system, from the broadest tenant levels down to specific data rows and columns. We also enabled audit attributes to ensure data quality by documenting data sources, reasons for data collection, date and time of data collection, and more. In this discussion, we will outline our implementation approach, review the results, and highlight our “lessons learned.”

Presto: Optimizing Performance of SQL-on-Anything Engine

DataWorks Summit

Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Bloomberg, Comcast, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores. With the ever-growing list of connectors to new data sources such as Azure Blob Storage, Elasticsearch, Netflix Iceberg, Apache Kudu, and Apache Pulsar, recently introduced Cost-Based Optimizer in Presto must account for heterogeneous inputs with differing and often incomplete data statistics. This talk will explore this topic in detail as well as discuss best use cases for Presto across several industries. In addition, we will present recent Presto advancements such as Geospatial analytics at scale and the project roadmap going forward.

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

DataWorks Summit

Specialized tools for machine learning development and model governance are becoming essential. MlFlow is an open source platform for managing the machine learning lifecycle. Just by adding a few lines of code in the function or script that trains their model, data scientists can log parameters, metrics, artifacts (plots, miscellaneous files, etc.) and a deployable packaging of the ML model. Every time that function or script is run, the results will be logged automatically as a byproduct of those lines of code being added, even if the party doing the training run makes no special effort to record the results. MLflow application programming interfaces (APIs) are available for the Python, R and Java programming languages, and MLflow sports a language-agnostic REST API as well. Over a relatively short time period, MLflow has garnered more than 3,300 stars on GitHub , almost 500,000 monthly downloads and 80 contributors from more than 40 companies. Most significantly, more than 200 companies are now using MLflow. We will demo MlFlow Tracking , Project and Model components with Azure Machine Learning (AML) Services and show you how easy it is to get started with MlFlow on-prem or in the cloud.

Extending Twitter's Data Platform to Google Cloud

DataWorks Summit

Twitter's Data Platform is built using multiple complex open source and in house projects to support Data Analytics on hundreds of petabytes of data. Our platform support storage, compute, data ingestion, discovery and management and various tools and libraries to help users for both batch and realtime analytics. Our DataPlatform operates on multiple clusters across different data centers to help thousands of users discover valuable insights. As we were scaling our Data Platform to multiple clusters, we also evaluated various cloud vendors to support use cases outside of our data centers. In this talk we share our architecture and how we extend our data platform to use cloud as another datacenter. We walk through our evaluation process, challenges we faced supporting data analytics at Twitter scale on cloud and present our current solution. Extending Twitter's Data platform to cloud was complex task which we deep dive in this presentation.

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

DataWorks Summit

At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

DataWorks Summit

Companies are increasingly moving to the cloud to store and process data. One of the challenges companies have is in securing data across hybrid environments with easy way to centrally manage policies. In this session, we will talk through how companies can use Apache Ranger to protect access to data both in on-premise as well as in cloud environments. We will go into details into the challenges of hybrid environment and how Ranger can solve it. We will also talk through how companies can further enhance the security by leveraging Ranger to anonymize or tokenize data while moving into the cloud and de-anonymize dynamically using Apache Hive, Apache Spark or when accessing data from cloud storage systems. We will also deep dive into the Ranger’s integration with AWS S3, AWS Redshift and other cloud native systems. We will wrap it up with an end to end demo showing how policies can be created in Ranger and used to manage access to data in different systems, anonymize or de-anonymize data and track where data is flowing.

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

DataWorks Summit

Advanced Big Data Processing frameworks have been proposed to harness the fast data transmission capability of Remote Direct Memory Access (RDMA) over high-speed networks such as InfiniBand, RoCEv1, RoCEv2, iWARP, and OmniPath. However, with the introduction of the Non-Volatile Memory (NVM) and NVM express (NVMe) based SSD, these designs along with the default Big Data processing models need to be re-assessed to discover the possibilities of further enhanced performance. In this talk, we will present, NRCIO, a high-performance communication runtime for non-volatile memory over modern network interconnects that can be leveraged by existing Big Data processing middleware. We will show the performance of non-volatile memory-aware RDMA communication protocols using our proposed runtime and demonstrate its benefits by incorporating it into a high-performance in-memory key-value store, Apache Hadoop, Tez, Spark, and TensorFlow. Evaluation results illustrate that NRCIO can achieve up to 3.65x performance improvement for representative Big Data processing workloads on modern data centers.

Computer Vision: Coming to a Store Near You

DataWorks Summit

Background: Some early applications of Computer Vision in Retail arose from e-commerce use cases - but increasingly, it is being used in physical stores in a variety of new and exciting ways, such as: ● Optimizing merchandising execution, in-stocks and sell-thru ● Enhancing operational efficiencies, enable real-time customer engagement ● Enhancing loss prevention capabilities, response time ● Creating frictionless experiences for shoppers Abstract: This talk will cover the use of Computer Vision in Retail, the implications to the broader Consumer Goods industry and share business drivers, use cases and benefits that are unfolding as an integral component in the remaking of an age-old industry. We will also take a ‘peek under the hood’ of Computer Vision and Deep Learning, sharing technology design principles and skill set profiles to consider before starting your CV journey. Deep learning has matured considerably in the past few years to produce human or superhuman abilities in a variety of computer vision paradigms. We will discuss ways to recognize these paradigms in retail settings, collect and organize data to create actionable outcomes with the new insights and applications that deep learning enables. We will cover the basics of object detection, then move into the advanced processing of images describing the possible ways that a retail store of the near future could operate. Identifying various storefront situations by having a deep learning system attached to a camera stream. Such things as; identifying item stocks on shelves, a shelf in need of organization, or perhaps a wandering customer in need of assistance. We will also cover how to use a computer vision system to automatically track customer purchases to enable a streamlined checkout process, and how deep learning can power plausible wardrobe suggestions based on what a customer is currently wearing or purchasing. Finally, we will cover the various technologies that are powering these applications today. Deep learning tools for research and development. Production tools to distribute that intelligence to an entire inventory of all the cameras situation around a retail location. Tools for exploring and understanding the new data streams produced by the computer vision systems. By the end of this talk, attendees should understand the impact Computer Vision and Deep Learning are having in the Consumer Goods industry, key use cases, techniques and key considerations leaders are exploring and implementing today.

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

DataWorks Summit

Whole genome shotgun based next generation transcriptomics and metagenomics studies often generate 100 to 1000 gigabytes (GB) sequence data derived from tens of thousands of different genes or microbial species. De novo assembling these data requires an ideal solution that both scales with data size and optimizes for individual gene or genomes. Here we developed an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions the reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomics and metagenomics test datasets from both short read and long read sequencing technologies. It achieved a near linear scalability with respect to input data size and number of compute nodes. SpaRC can run on different cloud computing environments without modifications while delivering similar performance. In summary, our results suggest SpaRC provides a scalable solution for clustering billions of reads from the next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar big data genomics problems.

More from DataWorks Summit (20)

Data Science Crash Course

Floating on a RAFT: HBase Durability with Apache Ratis

Tracking Crime as It Occurs with Apache Phoenix, Apache HBase and Apache NiFi

HBase Tales From the Trenches - Short stories about most common HBase operati...

Optimizing Geospatial Operations with Server-side Programming in HBase and Ac...

Managing the Dewey Decimal System

Practical NoSQL: Accumulo's dirlist Example

HBase Global Indexing to support large-scale data ingestion at Uber

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Building the High Speed Cybersecurity Data Pipeline Using Apache NiFi

Supporting Apache HBase : Troubleshooting and Supportability Improvements

Security Framework for Multitenant Architecture

Presto: Optimizing Performance of SQL-on-Anything Engine

Introducing MlFlow: An Open Source Platform for the Machine Learning Lifecycl...

Extending Twitter's Data Platform to Google Cloud

Event-Driven Messaging and Actions using Apache Flink and Apache NiFi

Securing Data in Hybrid on-premise and Cloud Environments using Apache Ranger

Big Data Meets NVM: Accelerating Big Data Processing with Non-Volatile Memory...

Computer Vision: Coming to a Store Near You

Big Data Genomics: Clustering Billions of DNA Sequences with Apache Spark

Recently uploaded

Full-RAG: A modern architecture for hyper-personalization

Zilliz

Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.

DevOps and Testing slides at DASA Connect

Kari Kakkonen

Pushing the limits of ePRTC: 100ns holdover for 100 days

Adtran

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

“I’m still / I’m still / Chaining from the Block”

Claudio Di Ciccio

20240607 QFM018 Elixir Reading List May 2024

Matthew Sinclair

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

Neo4j

UiPath Test Automation using UiPath Test Suite series, part 5

DianaGray10

みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...

名前です男

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Aggregage

Mind map of terminologies used in context of Generative AI

Kumud Singh

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...

Neo4j

Leonard Jayamohan, Partner & Generative AI Lead, Deloitte This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.

zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs

Alex Pruden

This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second). Paper: https://eprint.iacr.org/2023/1886

RESUME BUILDER APPLICATION Project for students

KAMESHS29

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Paige Cruz

Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack. I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Uni Systems S.M.S.A.

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

FIDO Alliance

Recently uploaded (20)

Full-RAG: A modern architecture for hyper-personalization

DevOps and Testing slides at DASA Connect

Pushing the limits of ePRTC: 100ns holdover for 100 days

Video Streaming: Then, Now, and in the Future

“I’m still / I’m still / Chaining from the Block”

20240607 QFM018 Elixir Reading List May 2024

Removing Uninteresting Bytes in Software Fuzzing

Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...

GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024

UiPath Test Automation using UiPath Test Suite series, part 5

Generative AI Deep Dive: Advancing from Proof of Concept to Production

Mind map of terminologies used in context of Generative AI

GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...

zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs

RESUME BUILDER APPLICATION Project for students

Essentials of Automations: The Art of Triggers and Actions in FME

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Uni Systems Copilot event_05062024_C.Vlachos.pdf

FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf

Introduction to Microsoft HDInsight and BI Tools

4. Introducing HDP for Windows OPERATIONAL DATA Hortonworks SERVICES SERVICES Data Platform (HDP) Manage & Manage & Store, Store, Operate at Operate at Process and Process and For Windows Scale Scale Access Data Access Data • 100% Open Source HADOOP CORE Distributed Enterprise Hadoop Storage & Processing • Component and version PLATFORM SERVICES Enterprise Readiness compatible with Microsoft HDInsight HORTONWORKS DATA PLATFORM (HDP) • Availability For Windows • Beta release available now • GA early 2Q 2013 Page 4 © Hortonworks Inc. 2013

5. Windows support in Apache Component Version Hadoop 1.0.3 Pig 0.9.3 Hive 0.9.0 HCatalog 0.4.1 WebHCat 0.1.4 Sqoop 1.4.2 Oozie 3.2.0 Initial Hadoop Core Apache JIRA: https://issues.apache.org/jira/browse/HADOOP-8079 Apache Hadoop branch-1-win github: https://github.com/apache/hadoop-common/tree/branch-1-win Apache Hadoop branch-trunk-win github: https://github.com/apache/hadoop-common/tree/branch-trunk-win © Hortonworks Inc. 2013

6. What do you get? • More deployment choices • Hadoop for Windows is for on-premise deployment – Good fit for organizations with Hadoop operational experience – Next step for those who are ready to move from POC to production • Use HDInsight for public and private cloud deployments – HDInsight Service -> Windows Azure– available in Preview today – HDInsight Server -> for interoperability across platforms of Hadoop, with Microsoft tools, on premise – Developer Preview available today • Full interoperability across platforms • Created through partnership between Hortonworks and Microsoft – Eighteen months of development time Page 6 © Hortonworks Inc. 2013

10. Seamless Interoperability with Your Microsoft Tools • Integrated with Microsoft APPLICATIONS tools for native big data analysis Microsoft Applications – Bi-directional connectors for SQL Server and SQL Azure through SQOOP – Excel ODBC integration through Hive • Addressing demand for DATA SYSTEMS HORTONWORKS DATA PLATFORM Hadoop on Windows For Windows – Ideal for Windows customers with Hadoop operational experience • Enables all common Hadoop workloads – Data refinement and ETL offload for DATA SOURCES high-volume data landing Traditional Sources New Sources (web logs, email, sensor data, social media) – Data exploration for discovery of new OLTP, (RDBMS, OLTP, OLAP) MOBILE POS DATA business opportunities SYSTEMS Page 10 © Hortonworks Inc. 2013

11. Hadoop and Microsoft Connectivity •Infrastructure Components • Active Directory • System Center •Microsoft Data Connectivity • Sqoop Connector • SQL Server / SQL Parallel Data Warehouse • Azure Storage / Azure Data Market •Microsoft Business Intelligence (BI) • ODBC Connectivity © Hortonworks Inc. 2013

15. Hadoop/Hive and Analysis Services •Delivers cost-efficient OLAP and data mining functionality •Supports ad-hoc data support queries in Microsoft Excel and dashboards •Offers exceptional query performance over vast amounts of data •SSAS to HIVE paper –http://msdn.microsoft.com/en- us/library/jj710329.aspx © Hortonworks Inc. 2013

16. Interoperability: Looking Ahead, PolyBase • Single query for relational & Hadoop data • Process data in place • Seamless: Regular T-SQL PolyBase command • Future expansion to other data sources © Hortonworks Inc. 2013

17. Agenda • HDP for Windows • Integrating with the Microsoft Tool Ecosystem • Demo • Q&A Page 17 © Hortonworks Inc. 2013

18. Demo Time! Excel integration with HDP • Interact with HDP through Excel • Use Data Explorer to explore and turn raw data into valuable information Page 18 © Hortonworks Inc. 2013

Introduction to Microsoft HDInsight and BI Tools

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Introduction to Microsoft HDInsight and BI Tools

Similar to Introduction to Microsoft HDInsight and BI Tools (20)

More from DataWorks Summit

More from DataWorks Summit (20)

Recently uploaded

Recently uploaded (20)

Introduction to Microsoft HDInsight and BI Tools