HTTPFS and Knox can be implemented together with Isilon OneFS to enhance HDFS access security in the following way:
1. HTTPFS acts as a gateway for HDFS, limiting direct access to HDFS ports and providing authentication. It must be configured for Kerberos if the Hadoop cluster uses Kerberos.
2. Knox integrates with HTTPFS and provides additional authorization, LDAP/AD integration, and perimeter security.
3. Together this solution provides a secure way to enable external WebHDFS access to HDFS stored on Isilon without exposing the Hadoop cluster directly. Firewalls can block direct access while still allowing controlled HDFS access via HTTPFS and Knox.
Here's a brief hands-on to get started with MySQL Server performance tuning. I'll show basic options to get started and have the right basics settings. Hope you'll enjoy!
Are you deploying Hadoop and want enterprise infrastructure manageability, reliability, and availability? The new EMC Hadoop Starter Kit shows you how to this without building HDFS data silo's.
Here's a brief hands-on to get started with MySQL Server performance tuning. I'll show basic options to get started and have the right basics settings. Hope you'll enjoy!
Are you deploying Hadoop and want enterprise infrastructure manageability, reliability, and availability? The new EMC Hadoop Starter Kit shows you how to this without building HDFS data silo's.
Overview of the architecture, and benefits of Dell HPC Storage with Intel EE Lustre in High Performance Computing and Big Science workloads.
Presented by Andrew Underwood at the Melbourne Big Data User Group - January 2016.
Lustre is a trademark of Seagate Technology.
Upgrade Without the Headache: Best Practices for Upgrading Hadoop in ProductionCloudera, Inc.
Walk through some of the best practices to keep in mind when it comes to upgrading your cluster, and learn how to leverage new Upgrade Wizard features in Cloudera Enterprise 5.3.
For most mission critical workloads, downtime is never an option. Any downtime can have a direct impact on revenue and lead to frantic calls in the middle of the night. For this reason, upgrading the software that powers these workloads can often be a daunting task. It can cause unpredictable issues without access to support. That’s why an enterprise-grade administration tool is crucial for running Hadoop in production. Hadoop consists of dozens of components, running across multiple machines, all with their own configurations. That can lead to a lot of complexity and uncertainty - especially when taking the upgrade plunge.
Cloudera Manager makes it easy and is the only production-ready administration tool for Hadoop. Not only does Cloudera Manager feature zero-downtime rolling upgrades, but it also has a built in Upgrade Wizard to make upgrades simple and predictable.
Deploying Apache Spark and testing big data applications on servers powered b...Principled Technologies
To get the most out of the heaps of data your company is sitting on, you’ll need a platform such as Apache Spark to sort through the noise and get meaningful conclusions you can use to improve your services. If you need to get results from such an intense workload in a reasonable amount of time, your company should invest in a solution whose power matches your level of work.
This proof of concept has introduced you to a new solution based on the AMD EPYC line of processors. Based on the new Zen architecture, the AMD EPYC line of processors offers resources and features worth considering. In the Principled Technologies datacenter, we set up a big data solution consisting of whitebox servers powered by the AMD EPYC 7601—the top-of-the-line offering from AMD. We ran an Apache Spark workload and tested the solution with three components of the HiBench benchmarking suite. The AMD systems maintained a consistent level of performance across these tests.
Managing Oracle Solaris Systems with Puppetglynnfoster
This presentation covers how to manage Oracle Solaris systems using Puppet. In this presentation we will cover the challenges facing the data center today, what Puppet is, and detail some of the work that was done to integrate Puppet with the core technology foundations included in the Oracle Solaris platform
Your datacenter is capable of doing great things—if you let it. Upgrades from Intel for compute, storage, and networking components can help your business support new services and expand your customer base. In our hands-on testing, we found that new Intel processors, high-bandwidth network components, and SATA or PCIe SSDs working together can boost your datacenter’s capabilities, which could translate to better business operations for your organization.
Spectrum Scale - Diversified analytic solution based on various storage servi...Wei Gong
This slides describe diversified analytic solutions based on Spectrum Scale with various deployment mode, such as storage rich-server, share storage, IBM DeepFlash 150 and Elastic Storage Server. It deep dives several advanced data management features and solutions for BD&A workload derived from Spectrum Scale.
Performance Comparison of Intel Enterprise Edition Lustre and HDFS for MapRed...inside-BigData.com
In this deck from the LAD'14 Conference in Reims, Rekha Singhal from Tata Consultancy Services presents: Performance Comparison of Intel Enterprise Edition Lustre and HDFS for MapReduce Application.
Learn more: http://insidehpc.com/lad14-video-gallery/
Watch the video presentation: http://inside-bigdata.com/2014/09/29/performance-comparison-intel-enterprise-edition-lustre-hdfs-mapreduce/
The Importance of Fast, Scalable Storage for Today’s HPCIntel IT Center
Today, data drives discovery. And discoveries create are key to creating sustained advantages. The better your critical workflows are able to create and access data – the better you’ll be able to discover new, innovative solutions to important problems, or to create entirely new products. More than ever before, data intensive applications need the sustained performance and virtually unlimited scalability that only parallel storage software delivers.
Designed for maximum performance and scale, storage solutions powered by Lustre software deliver the performance at scale to meet today’s storage requirements. As the most widely used parallel storage system for HPC, Lustre-powered storage is the ideal storage foundation.
But scalable performance storage by itself only solves half the problem. Today’s users expect storage solutions that deliver sustained performance, scale upward to near limitless capacities, and are simple to install and manage. Intel(r) Enterprise Edition for Lustre* software combines the straight line speed and scale of Lustre with the bottom line need for lowered management complexity and cost.
As the recognized leaders in the development and support of the Lustre file system, Intel has the expertise to make storage solutions for data intensive applications faster, smarter and easier.
Big-Data-as-a-Service (BDaaS) in an enterprise environment requires meeting the often contradictory goals of (1) providing your data scientists, analysts, and data engineers with a self-service consumption model; (2) delivering agile and scalable on-demand infrastructure for the rapidly evolving ecosystem of big data frameworks and application software; while (3) ensuring enterprise-grade capabilities for isolation, security, monitoring, etc.
In this presentation at our BDaaS meetup in Santa Clara, Tom Phelan (chief architect and co-founder of BlueData) reviewed these goals and how to resolve the potential contradictions. He also discussed the infrastructure, application, user experience, security, and maintainability considerations required before selecting (or designing and building) a Big-Data-as-a-Service platform for an enterprise big data deployment.
More info on this BDaaS meetup can be found at: http://www.meetup.com/Big-Data-as-a-Service/events/233999817
In this video from the LAD'14 Lustre Administrators and Developers Conference, Peter Jones from Intel presents: Lustre Releases.
Learn more: http://www.eofs.eu/?id=lad14
Watch the video presentation: http://wp.me/p3RLHQ-d1q
Symantec NetBackup 7.6 benchmark comparison: Data protection in a large-scale...Principled Technologies
The footprint of a VM can grow quickly in an enterprise environment and large-scale VM deployments in the thousands are common. As this number of deployed systems grows, so does the risk of failure. Critical failures can become unavoidable and offering data protection from a backup solution promotes business continuity. Elongated protection windows requiring multiple jobs of different types can create resource contention with production environments and may require valuable IT admin time, so a finite window for system backups can have plenty of importance.
In our hands-on SAN backup testing, the Symantec NetBackup Integrated Appliance running NetBackup 7.6 offered application protection to 1,000 VMs in 66.8 percent less time than Competitor “E” did. In addition, the Symantec NetBackup Integrated Appliance with NetBackup 7.6 created backup images that offered granular recovery without additional steps. These time and effort savings can scale as your VM footprint grows, allowing you to execute both system protection and user-friendly, simplified recovery.
Overview of the architecture, and benefits of Dell HPC Storage with Intel EE Lustre in High Performance Computing and Big Science workloads.
Presented by Andrew Underwood at the Melbourne Big Data User Group - January 2016.
Lustre is a trademark of Seagate Technology.
Upgrade Without the Headache: Best Practices for Upgrading Hadoop in ProductionCloudera, Inc.
Walk through some of the best practices to keep in mind when it comes to upgrading your cluster, and learn how to leverage new Upgrade Wizard features in Cloudera Enterprise 5.3.
For most mission critical workloads, downtime is never an option. Any downtime can have a direct impact on revenue and lead to frantic calls in the middle of the night. For this reason, upgrading the software that powers these workloads can often be a daunting task. It can cause unpredictable issues without access to support. That’s why an enterprise-grade administration tool is crucial for running Hadoop in production. Hadoop consists of dozens of components, running across multiple machines, all with their own configurations. That can lead to a lot of complexity and uncertainty - especially when taking the upgrade plunge.
Cloudera Manager makes it easy and is the only production-ready administration tool for Hadoop. Not only does Cloudera Manager feature zero-downtime rolling upgrades, but it also has a built in Upgrade Wizard to make upgrades simple and predictable.
Deploying Apache Spark and testing big data applications on servers powered b...Principled Technologies
To get the most out of the heaps of data your company is sitting on, you’ll need a platform such as Apache Spark to sort through the noise and get meaningful conclusions you can use to improve your services. If you need to get results from such an intense workload in a reasonable amount of time, your company should invest in a solution whose power matches your level of work.
This proof of concept has introduced you to a new solution based on the AMD EPYC line of processors. Based on the new Zen architecture, the AMD EPYC line of processors offers resources and features worth considering. In the Principled Technologies datacenter, we set up a big data solution consisting of whitebox servers powered by the AMD EPYC 7601—the top-of-the-line offering from AMD. We ran an Apache Spark workload and tested the solution with three components of the HiBench benchmarking suite. The AMD systems maintained a consistent level of performance across these tests.
Managing Oracle Solaris Systems with Puppetglynnfoster
This presentation covers how to manage Oracle Solaris systems using Puppet. In this presentation we will cover the challenges facing the data center today, what Puppet is, and detail some of the work that was done to integrate Puppet with the core technology foundations included in the Oracle Solaris platform
Your datacenter is capable of doing great things—if you let it. Upgrades from Intel for compute, storage, and networking components can help your business support new services and expand your customer base. In our hands-on testing, we found that new Intel processors, high-bandwidth network components, and SATA or PCIe SSDs working together can boost your datacenter’s capabilities, which could translate to better business operations for your organization.
Spectrum Scale - Diversified analytic solution based on various storage servi...Wei Gong
This slides describe diversified analytic solutions based on Spectrum Scale with various deployment mode, such as storage rich-server, share storage, IBM DeepFlash 150 and Elastic Storage Server. It deep dives several advanced data management features and solutions for BD&A workload derived from Spectrum Scale.
Performance Comparison of Intel Enterprise Edition Lustre and HDFS for MapRed...inside-BigData.com
In this deck from the LAD'14 Conference in Reims, Rekha Singhal from Tata Consultancy Services presents: Performance Comparison of Intel Enterprise Edition Lustre and HDFS for MapReduce Application.
Learn more: http://insidehpc.com/lad14-video-gallery/
Watch the video presentation: http://inside-bigdata.com/2014/09/29/performance-comparison-intel-enterprise-edition-lustre-hdfs-mapreduce/
The Importance of Fast, Scalable Storage for Today’s HPCIntel IT Center
Today, data drives discovery. And discoveries create are key to creating sustained advantages. The better your critical workflows are able to create and access data – the better you’ll be able to discover new, innovative solutions to important problems, or to create entirely new products. More than ever before, data intensive applications need the sustained performance and virtually unlimited scalability that only parallel storage software delivers.
Designed for maximum performance and scale, storage solutions powered by Lustre software deliver the performance at scale to meet today’s storage requirements. As the most widely used parallel storage system for HPC, Lustre-powered storage is the ideal storage foundation.
But scalable performance storage by itself only solves half the problem. Today’s users expect storage solutions that deliver sustained performance, scale upward to near limitless capacities, and are simple to install and manage. Intel(r) Enterprise Edition for Lustre* software combines the straight line speed and scale of Lustre with the bottom line need for lowered management complexity and cost.
As the recognized leaders in the development and support of the Lustre file system, Intel has the expertise to make storage solutions for data intensive applications faster, smarter and easier.
Big-Data-as-a-Service (BDaaS) in an enterprise environment requires meeting the often contradictory goals of (1) providing your data scientists, analysts, and data engineers with a self-service consumption model; (2) delivering agile and scalable on-demand infrastructure for the rapidly evolving ecosystem of big data frameworks and application software; while (3) ensuring enterprise-grade capabilities for isolation, security, monitoring, etc.
In this presentation at our BDaaS meetup in Santa Clara, Tom Phelan (chief architect and co-founder of BlueData) reviewed these goals and how to resolve the potential contradictions. He also discussed the infrastructure, application, user experience, security, and maintainability considerations required before selecting (or designing and building) a Big-Data-as-a-Service platform for an enterprise big data deployment.
More info on this BDaaS meetup can be found at: http://www.meetup.com/Big-Data-as-a-Service/events/233999817
In this video from the LAD'14 Lustre Administrators and Developers Conference, Peter Jones from Intel presents: Lustre Releases.
Learn more: http://www.eofs.eu/?id=lad14
Watch the video presentation: http://wp.me/p3RLHQ-d1q
Symantec NetBackup 7.6 benchmark comparison: Data protection in a large-scale...Principled Technologies
The footprint of a VM can grow quickly in an enterprise environment and large-scale VM deployments in the thousands are common. As this number of deployed systems grows, so does the risk of failure. Critical failures can become unavoidable and offering data protection from a backup solution promotes business continuity. Elongated protection windows requiring multiple jobs of different types can create resource contention with production environments and may require valuable IT admin time, so a finite window for system backups can have plenty of importance.
In our hands-on SAN backup testing, the Symantec NetBackup Integrated Appliance running NetBackup 7.6 offered application protection to 1,000 VMs in 66.8 percent less time than Competitor “E” did. In addition, the Symantec NetBackup Integrated Appliance with NetBackup 7.6 created backup images that offered granular recovery without additional steps. These time and effort savings can scale as your VM footprint grows, allowing you to execute both system protection and user-friendly, simplified recovery.
How to Protect Big Data in a Containerized EnvironmentBlueData, Inc.
Every enterprise spends significant resources to protect its data. This is especially true in the case of big data, since some of this data may include sensitive or confidential customer and financial information. Common methods for protecting data include permissions and access controls as well as the encryption of data at rest and in flight.
The Hadoop community has recently rolled out Transparent Data Encryption (TDE) support in HDFS. Transparent Data Encryption refers to the process whereby data is transparently encrypted by the big data application writing the data; it is not decrypted again until it is accessed by another application. The data is encrypted during its entire lifespan—in transit and at rest—except when it is being specifically accessed by a processing application.
TDE is an excellent approach for protecting data stored in data lakes built on the latest versions of HDFS. However, it does have its challenges and limitations. Systems that want to use TDE require tight integration with enterprise-wide Kerberos Key Distribution Center (KDC) services and Key Management Systems (KMS). This integration isn’t easy to set up or maintain. These issues can be even more challenging in a virtualized or containerized environment where one Kerberos realm may be used to secure the big data compute cluster and a different Kerberos realm may be used to secure the HDFS filesystem accessed by this cluster.
BlueData has developed significant expertise in configuring, managing, and optimizing access to TDE-protected HDFS. This session at the Strata Data Conference in March 2018 (by Thomas Phelan, co-founder and chief architect at BlueData) offers a detailed overview of how transparent data encryption works with HDFS, with a particular focus on containerized environments.
You’ll learn how HDFS TDE is configured and maintained in an environment where many big data frameworks run simultaneously (e.g., in a hybrid cloud architecture using Docker containers). Moreover, you’ll learn how KDC credentials can be managed in a Kerberos cross-realm environment to provide data scientists and analysts with the greatest flexibility in accessing data while maintaining complete enterprise-grade data security.
https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/63763
В связи с ростом трафика и необходимостью объемного анализа данных, большие данные стали одной из самых популярных областей в сфере IT, и многие компании в настоящее время работают над этим вопросом — развертывают кластеры проекта Hadoop, который в настоящее время является самой популярной платформой для обработки больших данных. В докладе в доступной форме будут представлены вопросы обеспечения безопасности Hadoop или, точнее, их принципы, а также продемонстрированы различные векторы атак на кластер.
A tutorial presentation based on hadoop.apache.org documentation.
I gave this presentation at Amirkabir University of Technology as Teaching Assistant of Cloud Computing course of Dr. Amir H. Payberah in spring semester 2015.
Accompanying slides for the class “Introduction to Hadoop” at the PRACE Autumn school 2020 - HPC and FAIR Big Data organized by the faculty of Mechanical Engineering of the University of Ljubljana (Slovenia).
Modernizing Your Data Platform for Analytics and AI in the Hybrid Cloud EraAlluxio, Inc.
Alluxio Webinar
August 25, 2021
For more Alluxio events: https://www.alluxio.io/events/
Speakers:
Adit Madan
Bin Fan
With data lakes expanding from on-prem to the cloud as well as increasing use of new object data stores, data platform teams are challenged with providing consistent, high-throughput access to distributed data sources for analytics and AI/ML applications. In today’s hybrid cloud and multi-cloud era, data-intensive applications such as Presto, Spark, Hive, and Tensorflow are suffering more sluggish response times and increased complexity with the growing separation of data and compute.
Join Alluxio’s distributed systems experts as they explore today’s data access challenges and open source data orchestration solutions for modernizing your data platform.
In this tech talk, you'll learn:
- How data access and throughput challenges are hindering large-scale analytics and AI/ML applications
- How a data orchestration layer can simplify distributed data access and improve performance
- Real-world production use cases and example journeys for architecting a modern data platform
Please explain in steps how to Add an index-html file and how to Uploa.pdfa1salesagency
Please explain in steps how to Add an index.html file and how to Upload an index.html file to
the AWS Linux server as asked below:
1. Read the success message - it will have info for the web site administrator (which is you)
telling you what directory to add content (/var/www/html). Add an index.html file to this
location. The index.html file should include your name and any other message you want to
include. Note that you will need to use sudo for any command that attempts to add or change a
file in the specified directory. There are a couple of ways you can do this:
a. Upload an index.html file to your AWS Linux server and copy or move it to the specified
location. Remember to move this file you must use the sudo command, for example "sudo mv
index.html /var/www/html"
b. Create a new index.html using any of the methods you have learned. Using the nano editor is
probably the easiest..
20+ Million Records a Second - Running Kafka on Isilon F800 Boni Bruno
This paper describes performance test results for running Kafka with Dell EMC Isilon F800 All-Flash NAS Storage. A comparison against direct attached storage is also provided.
Hadoop Tiering with Dell EMC Isilon - 2018Boni Bruno
Deep dive into HDFS Tiering with Dell EMC Isilon for Hadoop/Big Data. Covers MapReduce, Hive, and Spark use cases. Also incldues TPCDS Performance comparisons between Direct Attached Storage and Isilon Scale-out NAS Gen5 and Gen 6 models.
This presentation discusses the benefits of merging NPM & APM together to better assist problem response teams in troubleshooting network and application problems.
The presentation highlights a new product offering called NetPod which is a joint solution developed between Emulex and Dynatrace.
This presentation has been well received the the SANS community and many information security teams I engage with.
It describes how integrating a full content repository to your existing security architecture can decrease incident response time and lead to fast identification of root cause.
I also describe a new way of implementing NetFlow without sampling to provide greater visibility of your network.
Enjoy!
Boni Bruno, CISSP, CISM, CGEIT
www.bonibruno.com
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Explore our comprehensive data analysis project presentation on predicting product ad campaign performance. Learn how data-driven insights can optimize your marketing strategies and enhance campaign effectiveness. Perfect for professionals and students looking to understand the power of data analysis in advertising. for more details visit: https://bostoninstituteofanalytics.org/data-science-and-artificial-intelligence/
Data Centers - Striving Within A Narrow Range - Research Report - MCG - May 2...pchutichetpong
M Capital Group (“MCG”) expects to see demand and the changing evolution of supply, facilitated through institutional investment rotation out of offices and into work from home (“WFH”), while the ever-expanding need for data storage as global internet usage expands, with experts predicting 5.3 billion users by 2023. These market factors will be underpinned by technological changes, such as progressing cloud services and edge sites, allowing the industry to see strong expected annual growth of 13% over the next 4 years.
Whilst competitive headwinds remain, represented through the recent second bankruptcy filing of Sungard, which blames “COVID-19 and other macroeconomic trends including delayed customer spending decisions, insourcing and reductions in IT spending, energy inflation and reduction in demand for certain services”, the industry has seen key adjustments, where MCG believes that engineering cost management and technological innovation will be paramount to success.
MCG reports that the more favorable market conditions expected over the next few years, helped by the winding down of pandemic restrictions and a hybrid working environment will be driving market momentum forward. The continuous injection of capital by alternative investment firms, as well as the growing infrastructural investment from cloud service providers and social media companies, whose revenues are expected to grow over 3.6x larger by value in 2026, will likely help propel center provision and innovation. These factors paint a promising picture for the industry players that offset rising input costs and adapt to new technologies.
According to M Capital Group: “Specifically, the long-term cost-saving opportunities available from the rise of remote managing will likely aid value growth for the industry. Through margin optimization and further availability of capital for reinvestment, strong players will maintain their competitive foothold, while weaker players exit the market to balance supply and demand.”
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
As Europe's leading economic powerhouse and the fourth-largest hashtag#economy globally, Germany stands at the forefront of innovation and industrial might. Renowned for its precision engineering and high-tech sectors, Germany's economic structure is heavily supported by a robust service industry, accounting for approximately 68% of its GDP. This economic clout and strategic geopolitical stance position Germany as a focal point in the global cyber threat landscape.
In the face of escalating global tensions, particularly those emanating from geopolitical disputes with nations like hashtag#Russia and hashtag#China, hashtag#Germany has witnessed a significant uptick in targeted cyber operations. Our analysis indicates a marked increase in hashtag#cyberattack sophistication aimed at critical infrastructure and key industrial sectors. These attacks range from ransomware campaigns to hashtag#AdvancedPersistentThreats (hashtag#APTs), threatening national security and business integrity.
🔑 Key findings include:
🔍 Increased frequency and complexity of cyber threats.
🔍 Escalation of state-sponsored and criminally motivated cyber operations.
🔍 Active dark web exchanges of malicious tools and tactics.
Our comprehensive report delves into these challenges, using a blend of open-source and proprietary data collection techniques. By monitoring activity on critical networks and analyzing attack patterns, our team provides a detailed overview of the threats facing German entities.
This report aims to equip stakeholders across public and private sectors with the knowledge to enhance their defensive strategies, reduce exposure to cyber risks, and reinforce Germany's resilience against cyber threats.
1. IMPLEMENTING HTTPFS & KNOX WITH ISILON
ONEFS TO ENHANCE HDFS ACCESS SECURITY
Boni Bruno, CISSP, CISM, CGEIT
Principal Solutions Architect
DELL EMC
ABSTRACT
This paper describes implementing HTTPFS and Knox together with Isilon
OneFS to enhance HDFS access security. This integrated solution has
been tested and certified with Hortonworks on HDP v2.4 and Isilon OneFS
v 8.0.0.3.
2. CONTENTS
Introduction...................................................................................................................................................................3
WebHDFS REST API....................................................................................................................................................3
WebHDFS Port Assignment in Isilon OneFS...............................................................................................................5
WebHDFS Examples with ISILON...............................................................................................................................5
WebHDFS Security Concerns .....................................................................................................................................8
HTTPFS...........................................................................................................................................................................9
Installing HTTPFS........................................................................................................................................................9
Configuring HTTPFS .................................................................................................................................................10
Configuring HTTPFS for Kerberos ............................................................................................................................13
Running and Stopping HTTPFS.................................................................................................................................19
Configuring HTTPFS Auto-Start................................................................................................................................19
Testing HTTPFS ........................................................................................................................................................22
Knox.............................................................................................................................................................................24
Installing Knox..........................................................................................................................................................24
Configuring Knox using Ambari ...............................................................................................................................24
Configuring Knox for LDAP.......................................................................................................................................26
Configuring Knox for Kerberos.................................................................................................................................28
Testing Knox and Isilon Impersonation Defense .....................................................................................................30
Final Comments.......................................................................................................................................................35
Appendix......................................................................................................................................................................37
Additional Testing Results .......................................................................................................................................38
3. INTRODUCTION
Hadoop provides a Java native API to support file system operations such as create, rename or delete files and
directories, open, read or write files, set permissions, etc. This is great for applications running within the Hadoop
cluster, but there may be use cases where an external application needs to make such file system operations on
files stored on HDFS as well. Hortonworks developed the WebHDFS REST API to support these requirements based
on standard REST functionalities. WebHDFS REST APIs support a complete File System / File Context interface for
HDFS.
WEBHDFS REST API
WEBHDFS IS BASED ON HTTP OPERATIONS LIKE GET, PUT, POST AND DELETE. WEBHDFS
OPERATIONS LIKE OPEN, GETFILESTATUS, LISTSTATUS ARE USING HTTP GET, OTHER
OPERATIONS LIKE CREATE, MKDIRS, RENAME, SETPERMISSIONS ARE RELYING ON HTTP
PUT. APPEND OPERATIONS ARE BASED ON HTTP POST, WHILE DELETE IS USING HTTP
DELETE. AUTHENTICATION CAN BE BASED ON USER NAME, QUERY PARAMETER (AS PART
OF THE HTTP QUERY STRING) OR IF SECURITY IS ENABLED, THROUGH KERBEROS.
Web HDFS is enabled in a Hadoop cluster by defining the following property in hdfs-site.xml:
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
If using Ambari, enable WebHDFS under the General Settings of HDFS as shown below:
4. When using Isilon as a centralized HDFS storage repository for a given Hadoop Cluster, all namenode and datanode
functions must be configured to run on Isilon for the entire Hadoop cluster. By design, WebHDFS needs access to
all nodes in the cluster. Before the WebHDFS interface on Isilon can be used by the Hadoop Cluster, you must
enable WebHDFS in the Protocol Settings for HDFS on the designated Access Zone in Isilon - this is easily done in
the OneFS GUI. In the example below, hdp24 is the HDFS Access Zone for the Hadoop Cluster. Note the check
mark next to ENABLE WebHDFS access.
It is not sufficient to just enable WebHDFS in Ambari. Isilon must also be configured with WebHDFS enabled so
end to end WebHDFS communication can work in the Hadoop cluster. If multiple Access Zones are defined on
Isilon, make sure to enable WebHDFS as needed on each access zone.
5. WEBHDFS PORT ASSIGNMENT IN ISILON ONEFS
All references to Hadoop host hdp24 in this document refer to a defined SmartConnect HDFS Access Zone on
Isilon. TCP Port 8082 is the port OneFS uses for WebHDFS. It is important that the hdfs-site.xml file in the Hadoop
Cluster reflect the correct port designation for HTTP access to Isilon. See Ambari screen shot below for reference.
WEBHDFS EXAMPLES WITH ISILON
Assuming the Hadoop cluster is up and running with Isilon and WebHDFS has been properly enabled for the
Hadoop cluster, we are ready to test WebHDFS. CURL is a great command line tool for transferring data using
various protocols, including HTTP/HTTPS. The examples below use curl to invoke the WebHDFS REST API available
in Isilon OneFS to conduction various file system operations. Again, all references to hdp24 used in the curl
commands below refer to the SmartConnect HDFS Access Zone on Isilon and not some edge node in the cluster.
GETTING FILE STATUS EXAMPLE
The screen shot above shows curl being used to connect to Isilon’s WebHDFS interface on port 8082, the
GETFILESTATUS operation is used as user hduser1 to retrieve info on the projects.txt file.
Note: The projects.txt file is a test file I created. It is not part of the Hortonworks software.
6. A web browser may also be used to get projects.txt file status from Isilon WebHDFS as shown below:
This is similar to executing hdfs dfs –ls /user/hduser1/projects.txt from a Hadoop client node n107 as shown
below:
This quick example shows the flexibility of using WebHDFS. It provides a simple way to execute Hadoop file system
operations by an external client that does not necessarily run on the Hadoop cluster itself. Let’s look at another
example.
CREATING A DIRECTORY EXAMPLE
Here the MKDIRS operation on a different client node n105 is used with PUT to create the directory /tmp/hduser
as user hduser1 on Isilon. We can tell by the true Boolean result the operation was successful. We can also check
the result by using hdfs to see the directory on Isilon as shown below:
7. OPEN A FILE EXAMPLE
In the example above, the OPEN operation is used with curl to display the text string “Knox HTTPFS Isilon Project”
within the /tmp/hduser1/project.txt file.
As shown before, a web browser can be used to access the file as well. Here a browser is configured to
automatically open text files in notepad, so accessing the WebHDFS API on Isilon as shown below will open the
contents of /tmp/hduser1/project.txt in notepad directly.
To validate the contents from within the cluster, we can use hdfs as show below:
8. I’m only scratching the surface with the examples above; there are various operations you can execute with
WebHDFS. You can easily use WebHDFS to append data to files, rename files or directories, create new files, etc.
See the Appendix for many more examples.
It should be apparent that WebHDFS provides a simple, standard way to execute Hadoop file system operations
with external clients that do not necessarily run within the Hadoop cluster itself.
WEBHDFS SECURITY CONCERNS
SOMETHING WORTH POINTING OUT WITH THE ABOVE EXAMPLES AND WITH WEBHDFS IN
GENERAL – CLIENTS ARE DIRECTLY ACCESSING THE NAMENODES AND DATANODES VIA
PREDEFINED PORTS. THIS CAN BE SEEN AS A SECURITY ISSUE FOR MANY ORGANIZATIONS
WANTING TO ENABLE EXTERNAL WEBHDFS ACCESS TO THEIR HADOOP INFRASTRUCTURE.
Many organizations do not want their Hadoop infrastructure accessed directly from external clients. As seen thus
far, external clients can use WebHDFS to directly access the actual ports namenodes and datanodes are listening
on in the Hadoop Cluster and leverage the WebHDFS REST API to conduct various file system operations. Although
firewalls can filter access from external clients, the ports are still being directly access. As a result, firewalls do not
prohibit the execution of various WebHDFS operations.
The solution to this issue, in many cases, is to enable Kerberos in the Hadoop cluster and deploy Secure REST API
Gateways that enforce strong authentication and access control to WebHDFS. The remainder of this document
focuses on using HTTPFS and Knox in conjunction with Isilon OneFS to provide a secure WebHDFS deployment with
Hadoop. A diagram of the secure architecture is shown below for reference.
9. HTTPFS
The introduction section of this document provides an overview of WebHDFS and demonstrates how the
WebHDFS REST APIs support a complete File System / File Context interface for HDFS. WebHDFS is efficient as it
streams data from each datanode and can support external clients like curl or web browsers to extend data access
beyond the Hadoop cluster.
Since WebHDFS needs access to all nodes in the cluster by design, WebHDFS inherently establishes a wider foot
print for HDFS access in a Hadoop cluster since clients can access HDFS over HTTP/HTTPS. To help minimize the
size of the foot print to clients, a gateway solution is needed that provides a similar File System / File Context
interface for HDFS, and this is where HTTPFS comes in to play.
HTTPFS is a service that provides a REST HTTP gateway supporting all HDFS File System operations (read and
write). HTTPFS can be used to provide a gateway interface, i.e. choke point, to Isilon and limit broad HDFS access
from external clients to the Hadoop cluster. HTTPFS can also be integrated with Knox to improve service level
authorization, LDAP & AD integration, and overall perimeter security. See the Knox section of this document for
more details. The remainder of this section covers the installation and configuration of HTTPFS with Isilon.
INSTALLING HTTPFS
HTTPFS can be installed on Ambari server or a worker node, for production deployments, deploying on a dedicated
worker node is a best practice.
To install HTTPFS: yum install hadoop-httpfs (Note: existing HWX repos are hadoop-httpfs aware)
Note: The HTTPFS service is a tomcat application that relies on having the Hadoop libraries and configuration
available, so make sure to install HTTPFS on an edge node that is being managed by Ambari.
After you install HTTPFS, the directories below will be created on the HTTPFS server:
/usr/hdp/2.x.x.x-x/hadoop-httpfs
/etc/hadoop-httpfs/conf
/etc/hadoop-httpfs/tomcat-deployment
10. CONFIGURING HTTPFS
If you change directories to /usr/hdp on your HTTPFS server and list the files there, you will see a directory with
the version number of your existing HDP release. Make note of it so you can set the current version for httpfs. Set
the version for current with the following command:
hdp-select set hadoop-httpfs 2.x.x.x-x (replace the x with your HDP release)
The installation of httpfs above deploys scripts which have some hardcoded values that need to be changed.
Adjust the /usr/hdp/current/hadoop-httpfs/sbin/httpfs.sh script:
#!/bin/bash
# Autodetect JAVA_HOME if not defined
if [ -e /usr/libexec/bigtop-detect-javahome ]; then
. /usr/libexec/bigtop-detect-javahome
elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then
. /usr/lib/bigtop-utils/bigtop-detect-javahome
fi
### Added to assist with locating the right configuration directory
export HTTPFS_CONFIG=/etc/hadoop-httpfs/conf
### Remove the original HARD CODED Version reference.
Next, you need to create the following symbolic links:
cd /usr/hdp/current/hadoop-httpfs
ln -s /etc/hadoop-httpfs/tomcat-deployment/conf conf
ln -s ../hadoop/libexec libexec
11. Like all the other Hadoop components, httpfs follows the use of *-env.sh files to control the startup environment.
Above, in the httpfs.sh script we set the location of the configuration directory, this configuration directory is used
to find and load the httpfs-env.sh file.
The httpfs-env.sh file needs to be modified as shown below:
# Add exports to control and set the Catalina directories for starting and finding the httpfs application
export CATALINA_BASE=/usr/hdp/current/hadoop-httpfs
export HTTPFS_CATALINA_HOME=/etc/hadoop-httpfs/tomcat-deployment
# Set a log directory that matches your standards
export HTTPFS_LOG=/var/log/hadoop/httpfs
# Set a tmp directory for httpfs to store interim files
export HTTPFS_TEMP=/tmp/httpfs
The default port for httpfs is TCP 14000. If you need to change the port for httpfs, add the following export to the
above httpfs-env.sh file on the HTTPFS server:
export HTTPFS_HTTP_PORT=<new_port>
In the Ambari web interface, add httpfs as a proxy user in core-site.xml in the HDFS > Configs > Advanced >
Custom core site section:
Note: If the properties that are referenced below do not already exist, do the following steps:
1. Click the Add Property link in the Custom core site area to open the Add Property window.
2. Add each value in the <name> part in the Key field.
3. Add each value in the <value> part in the Value field.
4. Click Add. Then click Save.
13. CONFIGURING HTTPFS FOR KERBEROS
Ambari does not automate the configuration of HTTPFS to support Kerberos. If your Hadoop cluster was secured
with Kerberos using Ambari, you will need to create some needed keytabs and modify the httpfs-site.xml before
HTTPFS will work in a secure Kerberos Hadoop cluster.
The following assumptions are made for this section on configuring HTTPFS for Kerberos:
1. HTTPFS has been installed, configured, and verified to be working prior to enabling Kerberos.
2. Kerberos was enabled using Ambari and an MIT KDC and Isilon is configured and verified for Kerberos.
Both httpfs and HTTP service principals must be created for HTTPFS if they do not already exist.
Create the httpfs and HTTP (see note below) principals:
kadmin: addprinc -randkey httpfs/fully.qualified.domain.name@EXAMPLE-REALM.COM
kadmin: addprinc -randkey HTTP/fully.qualified.domain.name@EXAMPLE-REALM.COM
Note: HTTP principal and keytab may already exist as this is typically needed for other Hadoop services in
a secure Kerberos Hadoop cluster deployment. HTTP must be in CAPITAL LETTERS.
Create the keytab files for both httpfs and HTTP (see note above) principals:
kadmin -q "ktadd -k /etc/security/keytabs/httpfs.service.keytab httpfs/fully.qualified.domain.name@EXAMPLE-REALM.COM"
kadmin -q "ktadd -k /etc/security/keytabs/spnego.service.keytab HTTP/fully.qualified.domain.name@EXAMPLE-REALM.COM"
Note: The spnego keytab above only needs to be created if it does not already exist on the node running HTTPFS.
Merge the two keytab files into a single keytab file:
14. ktutil: rkt /etc/security/keytabs/httpfs.service.keytab
ktutil: rkt /etc/security/keytabs/spnego.service.keytab
ktutil: wkt /etc/security/ketyabs/httpfs-http.service.keytab
ktutil: quit
The above will create a file named httpfs-http.service.keytab in /etc/security/keytabs.
Note: This keytab should be copied to the HTTPFS node.
Test that the merged keytab file works:
klist -kt /etc/security/keytabs/httpfs-http.service.keytab
The above command should list both hdfs and HTTP principals for the httpfs-http.service.keytab. Below is an
example output from a test cluster:
Change the ownership and permissions of the /etc/security/keytabs/httpfs-http.service.keytab file:
chown httpfs:hadoop /etc/security/keytabs/httpfs-http.service.keytab
chmod 400 /etc/security/keytabs/httpfs-http.service.keytab
Edit the HTTPFS server httpfs-site.xml configuration file in the HTTPFS configuration directory by setting the
following properties:
15. httpfs.authentication.type: kerberos
httpfs.hadoop.authentication.type: kerberos
httpfs.authentication.kerberos.principal: HTTP/<FQDN of HTTPFS host>@< YOUR-REALM.COM>
httpfs.authentication.kerberos.keytab: /etc/hadoop-httpfs/conf/httpfs-http.service.keytab
httpfs.hadoop.authentication.kerberos.principal: httpfs/<FQDN of HTTPFS host>@< YOUR-REALM.COM>
httpfs.hadoop.authentication.kerberos.keytab: /etc/security/keytabs/httpfs-http.service.keytab
httpfs.authentication.kerberos.name.rules: Use the value configured for 'hadoop.security.auth_to_local' in
Ambari's HDFS Configs under "Advanced Core-Site".
An example httpfs-site.xml is listed below, with the relevant Kerberos information highlighted in red:
<configuration>
<!-- HTTPFS proxy user setting -->
<property>
<name>httpfs.proxyuser.knox.hosts</name>
<value>*</value>
</property>
<property>
<name>httpfs.proxyuser.knox.groups</name>
<value>*</value>
</property>
<!-- HUE proxy user setting -->
<property>
<name>httpfs.proxyuser.hue.hosts</name>
<value>*</value>
19. RUNNING AND STOPPING HTTPFS
Executing httpfs is simple.
To start:
cd /usr/hdp/current/hadoop-httpfs/sbin
./httpfs.sh start
To stop:
./httpfs.sh stop
CONFIGURING HTTPFS AUTO-START
As the root user, create the following hadoop-httpfs script in /etc/init.d:
#!/bin/bash
hdp-select set hadoop-httpfs 2.x.x.x.x-x
# See how we were called.
case "$1" in
start)
/usr/hdp/current/hadoop-httpfs/sbin/httpfs.sh start
;;
stop)
/usr/hdp/current/hadoop-httpfs/sbin/httpfs.sh stop
;;
*)
echo $"Usage: $prog {start|stop|restart}"
esac
20. As root user:
chmod 755 /etc/init.d/hadoop-httpfs
chkconfig --add hadoop-httpfs
# Start Service
service hadoop-httpfs start
# Stop Service
service hadoop-httpfs stop
This method will run the service as the httpfs user. Ensure that the httpfs user has permissions to write to the log
directory /var/log/hadoop/httpfs. The correct permission settings are shown below:
Note: the httpfs user also needs to be created on Isilon. The httpfs user is a system account that gets created
during installation of httpfs. As with all other Hadoop server accounts, Isilon needs to have all service accounts
defined as a LOCAL PROVIDER in the appropriate HDFS Access Zone (e.g. hdp24) as shown below.
21. Create the httpfs user in the LOCAL HDFS Access Zone for your cluster in Isilon OneFS. Assign the httpfs user to
the hadoop primary group. Leave the httpfs account Disabled as shown above and below. The UID on Isilon does
not need to match the UID on the httpfs server.
22. TESTING HTTPFS
As seen in the introduction section of this document, the curl command is an excellent tool for testing WebHDFS;
the same is true for testing HTTPFS. The default port for httpfs is TCP PORT 14000. The tests below show how
HTTPFS and Isilon OneFS can be used together in a Hadoop cluster. The requests made on port 14000 on the
HTTPFS gateway are passed to Isilon. The HTTPFS gateway is configured for Kerberos as is the Isilon HDFS Access
Zone. The Kerberos configuration is optional, but recommended for production Hadoop deployments to improve
cluster security.
The testing below is with Kerberos enabled. So make sure you have obtained and cached an appropriate Kerberos
ticket-granting ticket before running the commands. Use klist to verify you have a ticket cached as shown below:
GETTING A USER’S HOME DIRECTORY EXAMPLE
The screen shot above shows curl being used to connect to the HTTPFS gateway on port14000, the
GETHOMEDIRECTORY operation is used on user hduser1 to retrieve the home directory info.
HTTP enables GSS-Negotiate authentication. It is primarily meant as a support for Kerberos5 authentication but
may be also used along with another authentication method. GSS-Negotiate is specified with the –-negotiate
option with curl and the –w defines what to display on stdout after a completed and successful operation.
23. LIST DIRECTORY EXAMPLE
The screen shot above shows curl being used to connect to the HTTPFS gateway on port14000, the LISTSTATUS
operation is used as user hduser1 to do a directory listing on /tmp/hduser1.
CREAT DIRECTORY EXAMPLE
The screen shot above shows curl being used to connect to the HTTPFS gateway on port14000, the MKDIRS
operation is used as user hduser1 to create the directory /tmp/hduser1/test. The Boolean result of true means
the command executed successfully.
We can verify the creation of the directory with the hdfs command as show below:
This concludes the HTTPFS installation, configuration, and testing section of this document. The next section
covers how to integrate Knox with HTTPFS and Isilon.
24. KNOX
Knox enables the integration of enterprise identity management solutions and numerous perimeter security
features for REST/HTTP access to Hadoop and provides perimeter security for Hadoop services. Knox currently
supports YARN, WebHCAT, Oozie, HBase, Hive, and WebHDFS Hadoop services. The focus of this paper is on the
WebHDFS Hadoop service only. Just like HTTPFS, Knox can be installed on Kerberized and Non-Kerberized
Hadoop clusters.
Knox by default uses WebHDFS to perform any HDFS operation, but it can also leverage HTTPFS for the same HDFS
operations. Knox with HTTPFS provides a defense in depth strategy around REST/HTTP access to Hadoop and Isilon
OneFS.
This section covers the installation and configuration of Knox and LDAP services to work with HTTPFS in a
Kerberized cluster to provide secure REST/HTTP communications to Hadoop and Isilon OneFS.
INSTALLING KNOX
Knox is included with Hortonworks Data Platform by default. If you unselected the Knox service during installation
of HDP, just click the Actions button in Ambari and select the Knox service as shown below and click install.
CONFIGURING KNOX USING AMBARI
Knox can be managed through Ambari. Since HTTPFS runs on port 14000, a topology change to Knox for the
WebHDFS role is needed. Change the topology within the Advance topology section in Ambari/Knox, an example
topology configuration for the WebHDFS role is shown below:
25. The WebHDFS role is listed as a service in the topology configuration:
<service>
<role>WEBHDFS</role>
<url>http://<HTTPFS_HOST>:14000/webhdfs</url>
</service>
The HTTPFS_HOST should be replaced with the fully qualified name of the HTTPFS server. Port 14000 is the default
port for HTTPFS. If you made a change to the HTTPFS port assignment make sure to reflect the port change in the
Knox topology configuration as well. Everything else in the topology configuration can be left alone unless you
made other port changes to other services.
In the Ambari web interface, check that knox is configured as a proxy user in core-site.xml in the HDFS > Configs >
Advanced > Custom core site section and that the fully qualified domain name of the Knox host is set.
Note: If the properties that are referenced below do not already exist, do the following steps:
1. Click the Add Property link in the Custom core site area to open the Add Property window.
2. Add each value in the <name> part in the Key field.
3. Add each value in the <value> part in the Value field.
4. Click Add. Then click Save.
26. <property>
<name>hadoop.proxyuser.knox.host</name>
<value>n105.solarch.lab.emc.com</value>
</property>
<property>
<name>hadoop.proxyuser.knox.groups</name>
<value>users</value>
</property>
Make sure to restart HDFS and related components after making the above changes to core-site.xml.
CONFIGURING KNOX FOR LDAP
Knox can easily integrate with LDAP - just add an LDAP provider and associated parameters to the topology
configuration and you are done. An example LDAP provider (within the topology file) is shown below:
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapRealm.userDnTemplate</name>
28. CONFIGURING KNOX FOR KERBEROS
If the Hadoop cluster is secure with Kerberos, you need to make sure Knox is configured for Kerberos as well to
avoid authentication errors with the HTTPFS gateway and backend Isilon cluster. The Kerberos configuration for
Knox is done under Advance gateway-site in Ambari. An example configuration is shown below:
The Advanced gateway-site configuration allows you to specify the Knox gateway port (e.g. 8444), the location of
the krb5.conf (Kerberos configuration file), and set the gateway to use Kerberos (set to true).
The Advance knox-env in Ambari allows you to set the Knox user and group accounts, Knox keytab path, and Knox
Principal Name. An example configuration is shown below:
Note: the knox user also needs to be created on Isilon. The knox user is a system account that gets created during
installation of knox. As with all other Hadoop server accounts, Isilon needs to have all service accounts defined as
a LOCAL PROVIDER in the appropriate HDFS Access Zone (e.g. hdp24) as shown below.
29. Create the knox user in the LOCAL HDFS Access Zone for your cluster in Isilon OneFS. Assign the knox user to the
hadoop primary group. Leave the knox account Disabled as shown above and below. The UID on Isilon does not
need to match the UID on the knox server.
30. TESTING KNOX AND ISILON IMPERSONATION DEFENSE
Now that Knox and HTTPFS have been installed and configured, we can begin end-to-end testing with Isilon in a
secure Kerberos Hadoop cluster deployment using either curl or a web browser.
GETTING A USER’S HOME DIRECTORY EXAMPLE
The screen shot above shows curl being used to connect to the Knox gateway on port 8444 with LDAP user
ldapuser1, the GETHOMEDIRECTORY operation is used to retrieve the home directory info for the LDAP user. The
network connection to the Knox gateway is secured with TLS.
Let’s see what happens when we use the same REST HTTP operation over a web browser that connects to the
Knox gateway:
First, the Knox gateway will prompt for user authentication, after entering the correct LDAP credentials, we can
see the result of the REST HTTP GETHOMEDIRECTORY operation in the web browser as shown below:
31. Note that the network connection to the Knox gateway is secured with TLS as shown below:
I used self-signed certificates for this lab deployment, so there is a certificate error shown, but the network
connection is securely encrypted with TLS and a strong AES cipher.
OPENING A FILE EXAMPLE
Unlike the GETHOMEDIRECTORY operation shown in the previous test example, the OPEN operation actually
accesses data - we want to employ more security checks when data is being access in cases like this.
32. The screen shot above shows curl being used to connect to the Knox gateway on port 8444 as LDAP user
ldapuser1, the OPEN operation then tries to open the contents of the project.txt file in /tmp/hduser1, but a Server
Error is encountered. Although Isilon is aware of ldapuser1, Isilon provides an added layer of security to check for
impersonation attacks.
In this case, the HTTPFS gateway (which runs as the httpfs user) is acting as a proxy for user ldapuser1 REST HTTP
request between Knox and Isilon. When Isilon receives the OPEN request from httpfs on behalf of ldapuser1,
Isilon checks its Proxy User settings to see if httpfs is authorized to impersonate as ldapuser1 or the group
ldapuser1 is in, i.e. the hadoop group.
Assuming it is within policy for httpfs to impersonate anyone in the hadoop group, we can update the Proxy User
settings on Isilon so httpfs is authorized to process requests from either the ldapuser1 user specifically or anyone
in the hadoop group. The example below depicts a proxy configuration for the hadoop group:
With the proxy user setting in place, we can successfully run the previous test example to open a file:
33. As show above, with the correct Isilon Proxy User Policy in place on Isilon, the Open operation is now
allowed. Note: If the /tmp/hduser1 directory on Isilon did not have global read permissions set, this
operation would fail as shown below:
Changing the permissions on the /tmp/hduser1 directory on Isilon caused a permission denied error for
the same previous test operation. This is a testament to the embedded Isilon OneFS security features
and a benefit of using a centralized HDFS storage solution like Isilon.
CREAT DIRECTORY EXAMPLE
The screen shot above shows curl being used to connect to the Knox gateway on port 8444, the
MKDIRS operation is used as user ldapuser1 to create the directory /tmp/ldaptest. The Boolean result
of true means the command executed successfully.
34. We can verify the creation of the directory with the hdfs command as show below:
This concludes the Knox installation, configuration, and testing section of this document.
Please see the Appendix for additional Knox/HTTPFS/Isilon test examples.
35. FINAL COMMENTS
This solution has been tested and certified by both DELL EMC and Hortonworks with success. One thing that was
noticed during testing of the integrated solution is that httpfs wants the header “content-type: octet” stipulated
on data upload requests. The content-type is support by both WebHDFS & HTTPFS, but HTTPFS will throw a 400
Bad Request Error.
For example, let say you create a test data_file on the cluster with the CREATE operation, you will need to use the
–H flag with curl to specify the Content-Type accordingly, see example below:
With the Content-Type specified, the data upload successfully completes with no errors. This is an HTTPFS
requirement and has nothing to do with either Knox or Isilon OneFS. We can use hdfs command to see the
content of the created data_file as shown below:
36. Reading the file via curl does not require anything special as shown below:
The port for Knox was changed to 8444 instead of the default 8443. Be aware when setting up HTTPS for the
Ambari web interface, the default port is also 8443. To avoid port conflicts, I recommend you carefully assign a
unique port to your Knox gateway; port 8444 is a safe bet.
38. ADDITIONAL TESTING RESULTS
Below are additional testing examples for reference.
RENAMING A FILE EXAMPLE
The above curl command connects to the Knox gateway on port 8444 as LDAP user ldapuser1 to execute the
RENAME operation to rename data_file to data_file_new, the Boolean result of true means the command
executed successfully.
We can verify further my listing the contents of the /tmp/ldaptest directory:
SETTING FILE REPLICATION EXAMPLE
The above curl command connects to the Knox gateway on port 8444 as LDAP user ldapuser1 to execute
the SETREPLICATION operation to set replication to 1 for data_file_new, the Boolean result of true means
the command executed successfully.
39. Note: Isilon will always respond with true for these kinds of requests, but the reality is that Isilon OneFS file
system is much more efficient than HDFS, Isilon uses erasure encoding instead of replication to maintain
high availability.
SETTING FILE PERMISSIONS EXAMPLE
The above curl command connects to the Knox gateway on port 8444 as LDAP user ldapuser1 to execute
the SETPERMISSION operation to 777 for data_file_new, the HTTP/1.1 200 OK result of means the
command executed successfully. The hdfs command shows that the permissions for this data file were
changed on Isilon accordingly.
APPENDING DATA TO A FILE EXAMPLE
40. The above curl command connects to the Knox gateway on port 8444 as LDAP user ldapuser1 to execute
the APPEND operation to add ApendInfo to data_file_new, the HTTP/1.1 200 OK result of means the
command executed successfully. The hdfs command shows the data was appended successfully on Isilon.
RECURSIVE DELETE EXAMPLE
The above curl command connects to the Knox gateway on port 8444 as LDAP user ldapuser1 to execute
the DELETE operation to recursively delete from /tmp/ldaptest on, the HTTP/1.1 200 OK result of means the
command executed successfully. The hdfs command shows the directory and its content was successfully
removed from Isilon.