Horton works hdp 2.5 sandpit on microsoft azure v0 5
•Download as PPTX, PDF•
3 likes•870 views
Creating a Hortonworks Data Platform 2.5 Sandbox on Microsoft Azure
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Horton works hdp 2.5 sandpit on microsoft azure v0 5
Similar to Horton works hdp 2.5 sandpit on microsoft azure v0 5 (20)
Recently uploaded
Recently uploaded (20)
Horton works hdp 2.5 sandpit on microsoft azure v0 5
- 1. INSTALLING HORTONWORKS HDP 2.5 SANDBOX VM ON MICROSOFT AZURE Jeff Moss Oramoss Ltd
- 2. CONTENTS About Me HortonWorks Microsoft Azure Installation Of Hortonworks HDP 2.5 on Microsoft Azure Prerequisites Create VM Set Firewall Rules Change Passwords Access Ambari
- 3. ABOUT ME Independent Consultant Working with “big data” long before it became trendy! Extensive Oracle background 25 years experience from Oracle 5 onwards Former presenter UKOUG Currently Supplying Services To E.On UK Lead Technical Architect role 75TB Relational Enterprise Data Warehouse Support for several other European warehouses Contact Me http://www.oramoss.com http://www.oramoss.com/blog jeff.moss@oramoss.com
- 4. MICROSOFT AZURE Microsoft Cloud platform 2nd largest by share, behind Amazon (AWS) A collection of cloud services Most trusted cloud (compliance certifications) More geographical regions than Amazon Vendor Services Microsoft, Oracle, SAP… Open Source tools and technologies Linux, Docker, Chef, Puppet, GitLab, WordPress… Windows or Linux OS Big Data, Machine Learning, Blockchain, IoT, Microservices IAAS, PAAS, DBAAS, SAAS
- 5. HORTONWORKS HDP Commercial Hadoop Distribution One of the major distributions Partnership with Microsoft to deliver on Azure (Cloud)
- 6. HORTONWORKS HDP SANDBOX Prebuilt environment Virtualised VMWare VirtualBox Docker Useful for test and learn
- 7. PREREQUISITES Must Haves: A Microsoft Azure account I used the free month trial which gives £125 credit An SSH Client A Web Browser Optional (Not really) A generated public/private key pair
- 8. NAVIGATE TO THE PORTAL DASHBOARD
- 9. SEARCH FOR HORTONWORKS SANDBOX WITH HDP 2.5
- 10. SELECT THE HDP 2.5 VARIANT, PRESS CREATE
- 11. CONFIG STEP 1 – BASIC SETTINGS
- 12. CONFIG STEP 2 - SIZE
- 13. CONFIG STEP 3 - SETTINGS
- 14. CONFIG STEP 4 – SUMMARY PAGE
- 15. CONFIG STEP 5 – BUY
- 16. DEPLOYING
- 18. CLICK ON THE HDP25SANDBOX
- 19. ADD A DNS NAME LABEL
- 22. INBOUND SECURITY RULES Create additional rules for accessing optional components default-allow-4200 (Web SSH Access) default-allow-50070 (Default Name Node) default-allow-21000 (Atlas) default-allow-9995 (Zeppelin) default-allow-15000 (Falcon) default-allow-6080 (Ranger)
- 23. SSH ACCESS
- 24. CHANGE PASSWORDS Login as jeff sudo su – root ssh root@172.17.0.2 Set a new password for root (previously hadoop) Run ambari-admin-password-reset Set a new password for admin
- 25. HTTP ACCESS
- 28. WE’RE IN!
- 29. BACK TO HDP MAIN ENTRY
- 30. WEB BASED SSH ACCESS
- 31. WEB SSH TERMINAL
Editor's Notes
- Login to your Microsoft Azure account and got the Portal dashboard Click on MarketPlace
- In the search field enter Hortonworks and let it reduce the option to show the HDP 2.5 option and then click that.
- Ensure the 2.5 variant is selected on the left and click the Create button at the bottom right.
- Enter the name of the host you want to create Choose VM Disk Type of SSD (default) Enter a username – I’ve chosen jeff Choose either SSH Key or Password – I’ve not shown mine but I use a public/private SSH Key Pair Leave Subscription set to Free Trial (default) Choose Create New Resource Group and give it a name Choose a Location
- Choose a Machine type – I chose DS3_V2 Works out at about 26p per hour Click Select
- Use Managed Disks (Non Default) for simplicity Leave the defaults for all other settings and click OK
- Read the summary and click OK if you are happy
- Accept the terms by checking the check box and click Purchase.
- The VM will go off to create and you go back to the Portal page You can see that the VM is “deploying”, i.e. Azure is building it Eventually it finishes building – takes a couple of minutes
- After deploying it shows as “Running”
- Click on the VM and it takes you to the Summary page
- If it’s refreshed, from the Portal page Click on the Public IP Address from the Resources box, otherwise click on the VM, then the IP address, then Configuration. Set a DNS Name Label (the name you can use to access your VM on the internet) The Fully Qualified Domain Name is the Domain Name Label you choose with the suffix shown on the right hand side, e.g. oramosshdp25sandbox.ukwest.cloudapp.azure.com
- Navigate to Inbound Security Rules which is under the Network Security Group under the VM Notice the only rule that exists allows inbound SSH traffic. Click on Inbound security rule
- Click +Add in the top left to add a new rule to allow inbound traffic on port 8888 (for accessing HDP) Enter a name (allow-8888) and Port Range (8888) Click on OK to create the rule
- Now test you can access the site using your favourite terminal (I use MobaXterm) jeff@oramosshdp25sandbox.ukwest.cloudapp.azure.com In MobaXterm I set it up to use my private SSH key and the name of the user account (jeff) I specified earlier so it just logs in automatically without any further ado.
- In your SSH session follow these instructions to firstly set a new password for root – the default password is hadoop. Next as root, run ambari-admin-password-reset to enter a password for the ambari admin user (admin).
- Now navigate to the Fully Qualified Domain Name, port 8888. Fill out your details and hit Submit.
- Once you’ve entered your details it presents you with this screen Click on the Launch Dashboard on the left hand side which brings up the Ambari login page Log in as raj_ops / raj_ops
- Enter the admin user (admin) and the password you set a few moments ago with ambari-admin-password-reset. Alternatively use raj_ops / raj_ops which is the default Operator user already setup in the sandbox. Press Sign In
- That’s it…we’re up and running with Hortonworks Data Platform 2.5 in the Microsoft Azure cloud.
- This time, click on the Click Links on the right hand side which brings up the jump page for various tools
- Hover over SSH Client. Navigate to port 4200 on the main Fully Qualified Domain Name: http://oramosshdp25sandbox.ukwest.cloudapp.azure.com:4200/ Ambari – Systems Management environment for HortonWorks Atlas – Metadata management Falcon – Data lifecycle management – replication and lineage Ranger – Security Zeppelin – Web based notebook for ingestion and exploration
- Login using root and the password you have previously set.
- Login to hdfs user with su – hdfs Put out the hdfs admin report: hdfs dfsadmin -report
- hdfs dfs -ls / hdfs dfs -mkdir /jeffdir hdfs dfs -ls / echo "test" > test.txt hdfs dfs -put test.txt /jeffdir hdfs dfs -ls /jeffdir hdfs dfs -cat /jeffdir/test.txt hdfs dfs -rm /jeffdir/test.txt hdfs dfs -rmdir /jeffdir hdfs dfs -ls /