The document provides steps to install Java on an Ubuntu system and configure it so that it can be accessed from any location.
The key steps are: 1) Create a hadoop group and user, 2) Extract Java to /home/arun/bigdata/java, 3) Update alternatives to register Java, 4) Edit bash.bashrc to set JAVA_HOME and PATH, 5) Troubleshoot issues with file permissions and path errors. The goal is to properly configure the environment so that Java commands run smoothly from any folder.
Training on DSpace Institutional Repository
Organized by
BALID Institute of Information Management (BIIM
DSpace Manual for BALID Trainee
Institutional Repository
1-2 May 2014
Venue: CIRDAP
• Installation of DSpace on Debian
• Configuration of DSpace
• Customization of Dspace
• Cron Jobs setup for production system
• MTA Setup for DSpace
• Some Important Commands of PostgreSQL
• DSpace Discovery Setup
Prepared By
Nur Ahammad
Junior Assistant Librarian
Independent University, Bangladesh
Install and Configure Ubuntu for Hadoop Installation for beginners Shilpa Hemaraj
Covered each and every step to configure Ubuntu. Used vmware workstation 10.
Note: I am beginner so I might have used technical word wrong. But it is working perfectly fine.
Training on DSpace Institutional Repository
Organized by
BALID Institute of Information Management (BIIM
DSpace Manual for BALID Trainee
Institutional Repository
1-2 May 2014
Venue: CIRDAP
• Installation of DSpace on Debian
• Configuration of DSpace
• Customization of Dspace
• Cron Jobs setup for production system
• MTA Setup for DSpace
• Some Important Commands of PostgreSQL
• DSpace Discovery Setup
Prepared By
Nur Ahammad
Junior Assistant Librarian
Independent University, Bangladesh
Install and Configure Ubuntu for Hadoop Installation for beginners Shilpa Hemaraj
Covered each and every step to configure Ubuntu. Used vmware workstation 10.
Note: I am beginner so I might have used technical word wrong. But it is working perfectly fine.
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...Puppet
Here are the slides from Josh Snyder's presentation called Deconfiguration Management: Making Puppet Clean Up Its Own Mess. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa
Distribute cloud-environment-on-ubuntu-14-04-with-DockerLarry Lo
This is the basic tutorial to help developer or system administrator to build a basic cloud environment with Docker.
In this book, I will not use Dockerfile to create a container because I don't know how to use that yet. XDDD
In the end of this book, I will summary some problem I haven't solved yet.
If there is any mistake, please let me know.
GitBook:
Integrate Hue with your Hadoop cluster - Yahoo! Hadoop Meetupgethue
This talk will describe how Hue can be integrated with existing Hadoop deployments with minimal changes/disturbances. Romain will cover details on how Hue can leverage the existing authentication system and security model of your company. He will also cover the Hive/Shark/Pig/Oozie best practice setup for Hue.
http://www.meetup.com/hadoop/events/125191612/
Perl6 introduces a variety of tools for functional programming and writing readable code. Among them parameter declarations and lazy lists. This talk looks at how to get Perl6, where to find examples on RakudoCode, and how to use Perl6 functional tools for converting an algorithm from imperative to functional code, including parallel dispatch with the ">>" operator.
Introduction to command line tools for *NIX (UNIX (like OS X and Solaris/SunOS), BSD, & GNU/Linux) environments. I made this presentation originally for the LUG@UCF when I was an undergrad but still contains valid information. Hope you find it useful.
Hadoop installation on windows using virtual box and also hadoop installation on ubuntu
http://logicallearn2.blogspot.in/2018/01/hadoop-installation-on-ubuntu.html
Session 03 - Hadoop Installation and Basic CommandsAnandMHadoop
In this session you will learn:
Hadoop Installation and Commands
For more information, visit: https://www.mindsmapped.com/courses/big-data-hadoop/hadoop-developer-training-a-step-by-step-tutorial/
PuppetConf 2016: Deconfiguration Management: Making Puppet Clean Up Its Own M...Puppet
Here are the slides from Josh Snyder's presentation called Deconfiguration Management: Making Puppet Clean Up Its Own Mess. Watch the videos at https://www.youtube.com/playlist?list=PLV86BgbREluVjwwt-9UL8u2Uy8xnzpIqa
Distribute cloud-environment-on-ubuntu-14-04-with-DockerLarry Lo
This is the basic tutorial to help developer or system administrator to build a basic cloud environment with Docker.
In this book, I will not use Dockerfile to create a container because I don't know how to use that yet. XDDD
In the end of this book, I will summary some problem I haven't solved yet.
If there is any mistake, please let me know.
GitBook:
Integrate Hue with your Hadoop cluster - Yahoo! Hadoop Meetupgethue
This talk will describe how Hue can be integrated with existing Hadoop deployments with minimal changes/disturbances. Romain will cover details on how Hue can leverage the existing authentication system and security model of your company. He will also cover the Hive/Shark/Pig/Oozie best practice setup for Hue.
http://www.meetup.com/hadoop/events/125191612/
Perl6 introduces a variety of tools for functional programming and writing readable code. Among them parameter declarations and lazy lists. This talk looks at how to get Perl6, where to find examples on RakudoCode, and how to use Perl6 functional tools for converting an algorithm from imperative to functional code, including parallel dispatch with the ">>" operator.
Introduction to command line tools for *NIX (UNIX (like OS X and Solaris/SunOS), BSD, & GNU/Linux) environments. I made this presentation originally for the LUG@UCF when I was an undergrad but still contains valid information. Hope you find it useful.
Hadoop installation on windows using virtual box and also hadoop installation on ubuntu
http://logicallearn2.blogspot.in/2018/01/hadoop-installation-on-ubuntu.html
Session 03 - Hadoop Installation and Basic CommandsAnandMHadoop
In this session you will learn:
Hadoop Installation and Commands
For more information, visit: https://www.mindsmapped.com/courses/big-data-hadoop/hadoop-developer-training-a-step-by-step-tutorial/
To know more, Register for Online Hadoop Training at WizIQ.
Click here : http://www.wiziq.com/course/21308-hadoop-big-data-training
A complete guide to Hadoop Installation that will help you when ever you face problems while installing Hadoop !!
Get to know the configuration with Hadoop installation types and also handling of the HDFS files.
Let me know if anything is required. Happy to help.
Ping me google #bobrupakroy.
Talk soon!
install hadoop in windows using maven and windows sdk and visual c++ compiler.
To install hadoop on windows see below link step by step guidance.
From version 2.3 hadoop suppot windows also but by default it supports linux and other version. to install in windows need to compile the hadoop source in native windows sdk and then that hadoop distribution generated can be used to run hadoop in windows.
hadoop installation on windows
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
Unleashing the Power of Data_ Choosing a Trusted Analytics Platform.pdfEnterprise Wired
In this guide, we'll explore the key considerations and features to look for when choosing a Trusted analytics platform that meets your organization's needs and delivers actionable intelligence you can trust.
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...Social Samosa
The Modern Marketing Reckoner (MMR) is a comprehensive resource packed with POVs from 60+ industry leaders on how AI is transforming the 4 key pillars of marketing – product, place, price and promotions.
Analysis insight about a Flyball dog competition team's performanceroli9797
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
State of Artificial intelligence Report 2023kuntobimo2016
Artificial intelligence (AI) is a multidisciplinary field of science and engineering whose goal is to create intelligent machines.
We believe that AI will be a force multiplier on technological progress in our increasingly digital, data-driven world. This is because everything around us today, ranging from culture to consumer products, is a product of intelligence.
The State of AI Report is now in its sixth year. Consider this report as a compilation of the most interesting things we’ve seen with a goal of triggering an informed conversation about the state of AI and its implication for the future.
We consider the following key dimensions in our report:
Research: Technology breakthroughs and their capabilities.
Industry: Areas of commercial application for AI and its business impact.
Politics: Regulation of AI, its economic implications and the evolving geopolitics of AI.
Safety: Identifying and mitigating catastrophic risks that highly-capable future AI systems could pose to us.
Predictions: What we believe will happen in the next 12 months and a 2022 performance review to keep us honest.
Global Situational Awareness of A.I. and where its headedvikram sood
You can see the future first in San Francisco.
Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be un-leashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the wilful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.
Let me tell you what we see.
1. STEP 1:
copy necessary zip files to a folder like (/home/arun)
STEP 2:
arun@ubuntu:~/software$ sudo addgroup hadoop
[sudo] password for arun:
Adding group `hadoop' (GID 1001) ...
Done.
STEP 3:
Add user to the group hadoop so that only hadoop user in hadoop group can change the hadoop files
arun@ubuntu:~/software$ sudo adduser --ingroup hadoop hdadmin
Adding user `hdadmin' ...
Adding new user `hdadmin' (1001) with group `hadoop' ...
Creating home directory `/home/hdadmin' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for hdadmin
Enter the new value, or press ENTER for the default
Full Name []: arun
Room Number []: arun
Work Phone []: arun
Home Phone []: arun
Other []: arun
Is the information correct? [Y/n] y
2. STEP 4:
install java and copy to /home/arun/bigdata folder
NOTE: will cause link issue see below link and path cannot be same.
arun@ubuntu:~/software$ sudo tar vxzf jdk-7u79-linux-i586.tar.gz -C /home/arun/bigdata/java/
like in windows java,javaw,javaws are set in windows 32 path to make it accessible anywhere
ubuntu needs need to be triggered to make accessible from anywhere any folder. These three
applications files need to registered in ubuntu system by below commands.
$arun@ubuntu:~/software$ sudo update-alternatives --install "/usr/bin/java" java
"/home/arun/bigdata/java/jdk1.7.0_79" 1
$arun@ubuntu:~/software$ sudo update-alternatives --install "/usr/bin/javac" javac
"/home/arun/bigdata/java/jdk1.7.0_79/bin/javac" 1
$arun@ubuntu:~/software$ sudo update-alternatives --install "/usr/bin/javaws" javaws
"/home/arun/bigdata/java/jdk1.7.0_79/bin/javaws" 1
NOTE:
Make sure you give command correctly especially "/usr/bin/java" java part else it will override
sudo update-alternatives --install "/usr/bin/java" java "/home/arun/bigdata/java/jdk1.7.0_79" 1
sudo update-alternatives --install "/usr/bin/javac" javac
"/home/arun/bigdata/java/jdk1.7.0_79/bin/javac" 1
update-alternatives: using /home/arun/bigdata/java/jdk1.7.0_79/bin/javaws to provide
/usr/bin/javaws (javaws) in auto mode.
3. Each time you run it shows the below message indicating sucessfully registered
update-alternatives: renaming java link from /usr/bin/java to /usr/bin/javac.
arun@ubuntu:~/software$
if java installed in other than usr/bin folder these issue will occur.see below it adds extra /java
arun@ubuntu:~$ java -version
bash: /home/arun/bigdata/java/jdk1.7.0_79/bin/java: No such file or directory
arun@ubuntu:~$ java -version
bash: /home/arun/bigdata/java/jdk1.7.0_79/bin/java: No such file or directory
STEP 5:
update the environment variable using either nano, vi or gedit editor
root@ubuntu:/home/arun#nano /etc/bash.bashrc
^O to edit the file
^X to save the changes, it will ask whether to save the changes. if yes is given it will save.
Command:
arun@ubuntu:~$ gedit /etc/bash.bashrc
Append below lines to bash.bashrc file
export JAVA_HOME=/home/arun/bigdata/java/jdk1.7.0_79 or
4. export JAVA_HOME=/usr/lib/java/jdk1.7.0_79
export PATH=$PATH:$JAVA_HOME/bin
export is small case
since bash.bashrc file cannot be editted by normal user , use sudo root user in admin
arun@ubuntu:~$ sudo passwd root
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
go to root and edit.
arun@ubuntu:~$Sudo -i
the admin permission persist only for that session
arun@ubuntu:~$ su root
Password:
root@ubuntu:/home/arun#
To overcome the above error try Changing the path
$arun@ubuntu:~/software$ sudo update-alternatives --install
"/home/arun/bigdata/java/jdk1.7.0_79/bin/java" java "/home/arun/bigdata/java/jdk1.7.0_79" 1
$arun@ubuntu:~/software$ sudo update-alternatives –install
"/home/arun/bigdata/java/jdk1.7.0_79/bin/javac" javac
"/home/arun/bigdata/java/jdk1.7.0_79/bin/javac" 1
$arun@ubuntu:~/software$ sudo update-alternatives --install
"/home/arun/bigdata/java/jdk1.7.0_79/bin/javaws" javaws
"/home/arun/bigdata/java/jdk1.7.0_79/bin/javaws" 1
5. update-alternatives: <link> and <path> can't be the same
hence re-install java in /usr/java directory
To solve this issue
First create a dir in /usr/lib/
root@ubuntu:~# mkdir /usr/lib/java
or
root@ubuntu:~# sudo mkdir -p /usr/lib/java
arun@ubuntu:~/software$ sudo tar vxzf jdk-7u79-linux-i586.tar.gz -C
/usr/lib/java/
only in software dir only it allows to extract from root or arun dir it wont
allow to extract
root@ubuntu:/home/arun/ software# sudo tar vxzf jdk-7u79-linux-i586.tar.gz -C
/usr/lib/java/
now change the env variable
update the link:
$arun@ubuntu:~/software$ sudo update-alternatives --install "/usr/lib/java/jdk1.7.0_79/bin/java"
java "/usr/bin/java" 1
$arun@ubuntu:~/software$ sudo update-alternatives --install "/usr/lib/java/jdk1.7.0_79/bin/javac"
javac "/usr/bin/javac" 1
$arun@ubuntu:~/software$ sudo update-alternatives --install "/usr/lib/java/jdk1.7.0_79/bin/javaws"
javaws "/usr/bin/javaws" 1
6. Above see only javac,javaws is created during installation hence the error below
if any java,javac,javaws is not in /usr/bin/ better manually copy. Since below showing error
root@ubuntu:/# sudo update-alternatives --install "/usr/lib/java/jdk1.7.0_79/bin/java" java
"/usr/bin/java" 1
update-alternatives: error: alternative path /usr/bin/java doesn't exist.
Now try
7. $arun@ubuntu:~/software$ sudo update-alternatives --install "/usr/lib/java/jdk1.7.0_79/bin/java"
java "/usr/bin/java" 1
now worked...
sudo update-alternatives --install "/usr/lib/java/jdk1.7.0_79/bin/java" java "/usr/bin/java" 1
see javaws still pointing to /home /arun need re-installation just copy java,javac,jaws from
/usr/lib/java/bin to /usr/bin
root@ubuntu:/usr/bin# sudo update-alternatives --install "/usr/lib/java/jdk1.7.0_79/bin/java"
java "/usr/bin/java" 1
root@ubuntu:/usr/bin# sudo update-alternatives --install
"/usr/lib/java/jdk1.7.0_79/bin/javac" javac "/usr/bin/javac" 1
root@ubuntu:/usr/bin# sudo update-alternatives --install
"/usr/lib/java/jdk1.7.0_79/bin/javaws" javaws "/usr/bin/javaws" 1
update-alternatives: warning: forcing reinstallation of alternative
/home/arun/bigdata/java/jdk1.7.0_79/bin/javaws because link group javaws is broken.
To see which version is currently in use
arun@ubuntu:~$ update-alternatives --config java
arun@ubuntu:~$ update-alternatives --config javac
arun@ubuntu:~$ update-alternatives --config javaws
8. before install java:
arun@ubuntu:~$ update-alternatives --config java
update-alternatives: error: no alternatives for java.
Still no result
arun@ubuntu:~$ update-alternatives --config java
update-alternatives: error: no alternatives for java.
arun@ubuntu:~$
9. above vdi image alone is enough to create a new vdi from existing one. Easy transport this one to
any system.
10. If causing issues some issues then use it under /home/arun/bigdata/ without registering update-
alternatives showing error " /usr/bin/java No such file or directory"
export JAVA_HOME=/home/arun/bigdata/java/jdk1.7.0_79
export PATH=$PATH:$JAVA_HOME/bin
Three settings to note after installation of ubuntu
After installation of ubuntu and after guest installation cd images run check it is good
else problem with memory increase memory and try do not change any other settings.
STEP 1-Guest Additions
11. Once the guest additions installed become screen should be showed even compilatin fails is there
it is ok.Restart the system as mentioned and try.
STEP 2-Desktop Integration
Seamless mode is the correct mode to view once restart it should show correctly as below
i.e menuand header properly
12. NOTE: see below switch between normal seamless desktop integration mode. It should be
selected.Shortcut is (RIGHT+CTRL:L)
STEP 3 -Mouse Integration
see now mouse integration is checked
NOTE: enable host mouse pointer integration.
STEP 5 – Clipboard settings for copy from windows to ubuntu and viceversa
Check ubuntu-settings >general>advanced>shared clipboard (bi-directional)
drag-drop>bi-direction.
13. STEP 5 -Network
STEP 6 -Add user / group
add group
arun@ubuntu:~/java/jdk1.7.0_79/bin$ sudo addgroup hadoop
Adding group `hadoop' (GID 1001) ...
Done.
Add User
Adding user `hdadmin' ...
Adding new user `hdadmin' (1001) with group `hadoop' ...
Creating home directory `/home/hdadmin' ...
Copying files from `/etc/skel' ...
14. Reset Root passwd
arun@ubuntu:~/java$ sudo passwd root
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
arun@ubuntu:~/java$
login in to root passwd
STEP 7 – Check folder have permission
Try manually set environment variable and run java if wont run may be permission issue
hdadmin@ubuntu:/home/arun/java/jdk1.7.0_79/bin$ export
JAVA_HOME=/home/arun/java/jdk1.7.0_79
hdadmin@ubuntu:/home/arun/java/jdk1.7.0_79/bin$ export PATH=$PATH:
$JAVA_HOME/binhdadmin@ubuntu:/home/arun/java/jdk1.7.0_79/bin$ java -version
bash: /home/arun/java/jdk1.7.0_79/bin/java: No such file or directory
hdadmin@ubuntu:/home/arun/java/jdk1.7.0_79/bin$
15. it seems to be issue with the permission.
First login with a user created above and then install the java from that user login.
16. Use su root and delete old java as you can other cannot delete fully only su root can delete all files
login in now as created above user (hdadmin)
re-install java
17. NOTE:
hdadmin@ubuntu:/home/arun/softwares$ su hdadmin
Password:
hdadmin@ubuntu:/home/arun/softwares$
[sudo] password for hdadmin:
Sorry, try again.
[sudo] password for hdadmin:
hdadmin is not in the sudoers file. This incident will be reported.
Hduser is not in sudo list means not created user while installing ubuntu.
So need to add arun(user created at time of ubuntu installation) to the hadoop group then only we
can edit.
sudo usermod -a -G GROUPNAME USERNAME
root@ubuntu:/home/arun/softwares# usermod -a -G hadoop
arunroot@ubuntu:/home/arun/softwares# sudo tar vxzf jdk-7u79-linux-i586.tar.gz -C
/home/arun/java
jdk1.7.0_79/db/RELEASE-NOTES.html
jdk1.7.0_79/db/bin/
jdk1.7.0_79/db/bin/setNetworkClientCP
18. jdk1.7.0_79/db/bin/startNetworkServer
Change owner of the jdk:
chown -R <<user>>:<<group>> <<folder>>
root@ubuntu:/home/arun/java# sudo chown -R arun:hadoop jdk1.7.0_79
root@ubuntu:/home/arun/java#
see the lock icon gone now.
Change all the folder beneath to share common hadoop group permission
19. root@ubuntu:/home/arun/java# sudo chown -R arun:hadoop jdk1.7.0_79/bin/*
root@ubuntu:/home/arun/java#
root@ubuntu:/home/arun/java#sudo chmod 777
still no such file or directory
arun@ubuntu:~$ java -version
bash: /home/arun/java/jdk1.7.0_79/bin/java: No such file or directory
You get this error because you try to run a 32-bit executable on a 64-bit operating system.
And the message No such file or directory does not refer to your executable file called a.
Instead the error it refers to a helper program that's needed to run the 32-bit dynamically linked
executable.
After installation to jdk 64 version it is working fine!!!
20. chown -R <<user>>:<<group>> <<folder>>
root@ubuntu:/home/arun/hadoop# sudo chown -R arun:hadoop ./
change the permission of the folder hadoop-2.5.0 and other sub-folder beneath to hadoop
group.as you can see below -lock icon is now not there nomore when comparing the folder above.
22. Enter file in which to save the key (/home/arun/.ssh/id_rsa): hadoop
Your identification has been saved in hadoop.
Your public key has been saved in hadoop.pub.
The key fingerprint is:
5c:a3:5a:fe:d1:95:ed:1b:a0:ef:95:7c:d7:d8:f3:b5 arun@ubuntu
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| o |
| . o . o |
| S .o .|
| + ...o+o|
| . . ... .*B|
| . .. . X|
| . .o E.|
+-----------------+
arun@ubuntu:~/putty/putty-0.66$
>press CTRL+H to view the hidden files
As you can see it is stored in hadoop .pub file in inside putty.
23. Create a folder hadoop key and Set the key as trusted key for remote login
hadoopkey does not get created with any file. Change the /etc/sysctl.conf files to allow
connection to establish.
24. You have to reboot your machine in order to make the changes take effect.
You can check whether IPv6 is enabled on your machine with the following command:
$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
0
A return value of 0 means IPv6 is enabled, a value of 1 means disabled (that’s what we want).
25. Reboot the ubuntu
Try check the command again.
$ cat /proc/sys/net/ipv6/conf/all/disable_ipv6
0
if still not working change manually
root@ubuntu:/home/arun# nano /proc/sys/net/ipv6/conf/all/disable_ipv6
1
root@ubuntu:/home/arun/putty/putty-0.66# ssh localhost
ssh: connect to host localhost port 22: Connection refused
root@ubuntu:/home/arun/putty/putty-0.66# sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
root@ubuntu:/home/arun/putty/putty-0.66# ssh localhost
ssh: connect to host localhost port 22: Connection refused
root@ubuntu:/home/arun/putty/putty-0.66# sudo iptables -A INPUT -p tcp --dport ssh -j ACCEPT
root@ubuntu:/home/arun/putty/putty-0.66# ssh localhost
ssh: connect to host localhost port 22: Connection refused
root@ubuntu:/home/arun/putty/putty-0.66#
26. create a folder inside /home/arun/putty/putty-0.66/hadoopkey
where hadoopkey – newly created folder to copy the key
go inside the hadoop key and allow the key to be accepted as trustedkey.
arun@ubuntu:~/putty/putty-0.66/hadoopkey$ cat hadoop.pub >>authorized_keys
arun@ubuntu:~/putty/putty-0.66/hadoopkey$
27. check hadoop.pub is working and has got generated with key
then Set the key as trusted key for remote login:
arun@ubuntu:~/putty/putty-0.66/hadoopkey$ cat hadoop.pub >>authorized_keys
arun@ubuntu:~/putty/putty-0.66/hadoopkey$
now below you can see the authorized_keys is get generated toallow hand shake to happen
28. full steps to connect to ssh:
Install SSH server:
$ sudo apt-get install openssh-server
if not worked download tar and then copy to the folder and try below commands
Create a ssh-key without password:
$ ssh-keygen -t rsa -P “”
Set the key as trusted key for remote login:
arun@ubuntu:~/putty/putty-0.66/hadoopkey$ cat hadoop.pub >>authorized_keys
Try to connect on localhost and accept the connection (mandatory)
$ ssh localhost
hadoop – file name under which key to be generated -inside folder /home/arun/putty/putty-
29. 0.66/hadoopkey/
arun@ubuntu:~/putty/putty-0.66$ ssh-keygen -t rsa -P “”
Generating public/private rsa key pair.
Enter file in which to save the key (/home/arun/.ssh/id_rsa): /home/arun/putty/putty-
0.66/hadoopkey/hadoop
/home/arun/putty/putty-0.66/hadoopkey/hadoop already exists.
Overwrite (y/n)? y
Your identification has been saved in /home/arun/putty/putty-0.66/hadoopkey/hadoop.
Your public key has been saved in /home/arun/putty/putty-0.66/hadoopkey/hadoop.pub.
The key fingerprint is:
42:fa:dd:7c:a3:22:2b:81:7f:69:c1:fd:c7:96:f7:1d arun@ubuntu
The key's randomart image is:
+--[ RSA 2048]----+
| |
| |
| . |
| o |
| .....S |
| . ..oo.o |
| . ..o..o.o. E |
| o = . .o=.. o|
| +.o ..o . .o|
+-----------------+
arun@ubuntu:~/putty/putty-0.66/hadoopkey$ cat hadoop.pub >>authorized_keys
arun@ubuntu:~/putty/putty-0.66/hadoopkey$
enable ssh port 22
30. uncomment this line and restart
below are the ssh commands and functionalities:
ssh(1) - The basic rlogin/rsh-like client program.
•sshd(8) - The daemon that permits you to login.
•ssh_config(5) - The client configuration file.
•sshd_config(5) - The daemon configuration file.
•ssh-agent(1) - An authentication agent that can store private keys.
•ssh-add(1) - Tool which adds keys to in the above agent.
•sftp(1) - FTP-like program that works over SSH1 and SSH2 protocol.
•scp(1) - File copy program that acts like rcp(1).
•ssh-keygen(1) - Key generation tool.
•sftp-server(8) - SFTP server subsystem (started automatically by sshd).
•ssh-keyscan(1) - Utility for gathering public host keys from a number of hosts.
•ssh-keysign(8) - Helper program for hostbased authentication.
31. Configuring OpenSSH
Config Files
~/.ssh/*, /etc/ssh/ssh_config, and /etc/ssh/sshd_config
There are no required changes to any of these files. However, you may wish to
view the /etc/ssh/ files and make any changes appropriate for the security of
your system. One recommended change is that you disable root login via ssh.
Execute the following command as the root user to disable root login via ssh:
echo "PermitRootLogin no" >> /etc/ssh/sshd_config
above command create a file sshd_config under /etc/ssh/
If you want to be able to log in without typing in your password, first create
~/.ssh/id_rsa and ~/.ssh/id_rsa.pub with ssh-keygen and then copy
~/.ssh/id_rsa.pub to ~/.ssh/authorized_keys on the remote computer that you
want to log into. You'll need to change REMOTE_USERNAME and
REMOTE_HOSTNAME for the username and hostname of the remote computer
and you'll also need to enter your password for the ssh-copy-id command to
succeed:
ssh-keygen &&
ssh-copy-id -i ~/.ssh/id_rsa.pub REMOTE_USERNAME@REMOTE_HOSTNAME
ssh-keygen &&
ssh-copy-id -i /home/arun/sshserver/hadoopkey/hadoop.pub arun@localhost
Once you've got passwordless logins working it's actually more secure than
logging in with a password (as the private key is much longer than most
people's passwords). If you would like to now disable password logins, as
the root user:
echo "PasswordAuthentication no" >> /etc/ssh/sshd_config &&
32. echo "ChallengeResponseAuthentication no" >> /etc/ssh/sshd_config
If you added LinuxPAM support and you want ssh to use it then you will need
to add a configuration file for sshd and enable use of LinuxPAM. Note, ssh only
uses PAM to check passwords, if you've disabled password logins these
commands are not needed. If you want to use PAM issue the following
commands as the root user:
sed 's@d/login@d/sshd@g' /etc/pam.d/login > /etc/pam.d/sshd &&
chmod 644 /etc/pam.d/sshd &&
echo "UsePAM yes" >> /etc/ssh/sshd_config
Additional configuration information can be found in the man pages
for sshd, ssh and ssh-agent.
Boot Script
To start the SSH server at system boot, install the /etc/rc.d/init.d/sshd init
script included in the blfs-bootscripts-20150924 package.
make install-sshd
Contents
Installed Programs:scp, sftp, slogin (symlink to ssh), ssh, ssh-add, ssh-agent,
ssh-copy-id, ssh-keygen, ssh-keyscan, and sshd
Installed Libraries:None
Installed Directories:/etc/ssh, /usr/share/doc/openssh-7.1p1, and /var/lib/sshd
Short Descriptions
scp is a file copy program that acts like rcp except it uses an encrypted
protocol.
sftp is an FTP-like program that works over the SSH1 and SSH2 protocols.
slogin is a symlink to ssh.
ssh is an rlogin/rsh-like client program except it uses an encrypted protocol.
sshd is a daemon that listens for ssh login requests.
ssh-add is a tool which adds keys to the ssh-agent.
ssh-agent is an authentication agent that can store private keys.
ssh-copy-id is a script that enables logins on remote machine using local keys.
ssh-keygen is a key generation tool.
ssh-keyscan is a utility for gathering
34. Ubuntu install ssh manually:
root@ubuntu:/home/arun# ssh-keygen &&
> ssh-copy-id -i /home/arun/sshserver/hadoopkey/hadoop.pub arun@localhost
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): /home/arun/sshserver/hadoopkey/hadoop.pub
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
passphrase too short: have 4 bytes, need > 4
Saving the key failed: /home/arun/sshserver/hadoopkey/hadoop.pub.
root@ubuntu:/home/arun# ssh-keygen && ssh-copy-id -i
/home/arun/sshserver/hadoopkey/hadoop.pub arun@localhost
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): ^C
root@ubuntu:/home/arun# clear
root@ubuntu:/home/arun# ssh-keygen && ssh-copy-id -i
/home/arun/sshserver/hadoopkey/hadoop.pub arun@localhost
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): /home/arun/sshserver/hadoopkey/hadoop.pub
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
passphrase too short: have 2 bytes, need > 4
Saving the key failed: /home/arun/sshserver/hadoopkey/hadoop.pub.
root@ubuntu:/home/arun# ssh-keygen && ssh-copy-id -i
/home/arun/sshserver/hadoopkey/hadoop.pub arun@localhost
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): ^C
root@ubuntu:/home/arun# clear
root@ubuntu:/home/arun# ssh-keygen && ssh-copy-id -i
/home/arun/sshserver/hadoopkey/hadoop.pub arun@localhost
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): /home/arun/sshserver/hadoopkey/hadoop.pub
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/arun/sshserver/hadoopkey/hadoop.pub.
Your public key has been saved in /home/arun/sshserver/hadoopkey/hadoop.pub.pub.
The key fingerprint is:
d8:78:0e:96:07:f1:0d:29:62:aa:b3:68:90:42:ca:c4 root@ubuntu
The key's randomart image is:
+--[ RSA 2048]----+
35. | . .. |
| o .o.o |
|. o .... . |
| E. * |
|=o * S |
|B. . = |
|+o . |
|o. |
|. |
+-----------------+
ssh: connect to host localhost port 22: Connection refused
root@ubuntu:/home/arun#
create hadoop boot scripts
first extract the boot scripts after downloading from ssh site.
http://www.linuxfromscratch.org/blfs/view/svn/introduction/bootscripts.html
NOTE : for extract the tar file i.e not in format tar.gz it should be tar xvf <<fileName>>
36. try now create the boot script as per above step -copied below for
reference, then create boot script.
Boot Script
To start the SSH server at system boot, install the /etc/rc.d/init.d/sshd init
script included in the blfs-bootscripts-20150924 package.
make install-sshd
NOTE:
just download the boot script from site and follow the steps in makefile
37. TO install sshd :
just set values for this.
ETCDIR=/etc
LIBDIR=${DESTDIR}/lib/services
EXTDIR=${DESTDIR}${ETCDIR}
MODE=754
DIRMODE=755
CONFMODE=644
48. see from ipconfig virtual ipv4 address working.
arun@ubuntu:~$ ssh localhost
ssh: connect to host localhost port 22: Connection refused
arun@ubuntu:~$ ssh arun@localhost
ssh: connect to host localhost port 22: Connection refused
arun@ubuntu:~$ ssh arun@192.168.1.2
ssh: connect to host 192.168.1.2 port 22: Connection refused
arun@ubuntu:~$ ssh arun@192.168.1.2:80
ssh: Could not resolve hostname 192.168.1.2:80: Name or service not known
arun@ubuntu:~$ ssh arun@192.168.1.2
ssh: connect to host 192.168.1.2 port 22: Connection refused
arun@ubuntu:~$ ssh arun@192.168.1.2
^C
arun@ubuntu:~$ ssh arun@192.168.1.2
in ubuntu mostly dynamic ip is used if not configured to configure statically
NOTE :
in the above ipconfig
192.168.1.2 ipv4 config of windows ip only works 192.168.1.56 of virutal box wont work see
below screenshot vitual box ip not works!!!
49. after that check fire fox loading web pages see for me the above config works.
address should be same in the ipconfig in windows above. Also make sure the network adapter is
bridge adapter
To check the network interfaces
53. see still connection issue of ssh :
arun@ubuntu:~/putty/putty-0.66$ ssh-keygen -t rsa -P “”
Generating public/private rsa key pair.
Enter file in which to save the key (/home/arun/.ssh/id_rsa):
/home/arun/putty/putty-0.66/hadoopkey/hadoop
go to user folder in this case arun
54. check inside arun to check whether public and private are generated.
To check the hidden file (CTRL-H) or ls -la to get the hidden file info
arun@ubuntu:~/.ssh$ ls -la
total 16
drwx------ 2 arun arun 4096 Nov 21 08:35 .
drwxr-xr-x 29 arun arun 4096 Nov 21 08:35 ..
-rw------- 1 arun arun 1766 Nov 21 08:36 id_rsa
-rw-r--r-- 1 arun arun 393 Nov 21 08:36 id_rsa.pub
arun@ubuntu:~/.ssh$ ls
id_rsa -privatekey
id_rsa.pub- public key
56. hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop namenode -format
hduser@ubuntu:~$ /usr/local/hadoop/bin/start-all.sh
hduser@ubuntu:/usr/local/hadoop$ jps
to check ubuntu supported packages.
arun@ubuntu:~$ apt-cache search openssh
libssl1.0.0 - SSL shared libraries
ssh-askpass-gnome - interactive X program to prompt users for a passphrase for ssh-add
openssh-client - secure shell (SSH) client, for secure access to remote machines
arun@ubuntu:~$
NOTE: if you don't know the exact package you can use apt-cache search<<key>>
58. see since it is commented it is blank
IMPORTANT:
Type lsb_release -sc to find out your release. You may repeat the commands with "deb-src"
instead of "deb" in order to install the source files.
59. Adding Repositories
Adding the Universe and Multiverse Repositories
Additional software repositories such as Universe and Multiverse can be enabled by
uncommenting the corresponding apt lines (i.e. delete the '#' at the beginning of the line). For
Universe, uncomment those lines:
•deb http://us.archive.ubuntu.com/ubuntu/ saucy universe
deb-src http://us.archive.ubuntu.com/ubuntu/ saucy universe
deb http://us.archive.ubuntu.com/ubuntu/ saucy-updates universe
deb-src http://us.archive.ubuntu.com/ubuntu/ saucy-updates universe
There are four similar lines for 'multiverse'.
OR you may use the add-apt-repository command. If your release is 'saucy':
•sudo add-apt-repository "deb http://us.archive.ubuntu.com/ubuntu/ saucy universe
multiverse"
sudo add-apt-repository "deb http://us.archive.ubuntu.com/ubuntu/ saucy-updates
universe multiverse"
Depending on your location, you should replace 'us.' by another country code, referring to a
mirror server in your region. Check sources.list to see what is used!
Type lsb_release -sc to find out your release. You may repeat the commands with "deb-src"
instead of "deb" in order to install the source files.
Don't forget to retrieve the updated package lists:
•sudo apt-get update
Adding Partner Repositories
You can add the partner repositories by uncommenting the following lines in your
/etc/apt/sources.list file:
•deb http://archive.canonical.com/ubuntu saucy partner
deb-src http://archive.canonical.com/ubuntu saucy partner
Then update as before:
•sudo apt-get update
Be aware that the software contained within this repository is NOT open source.
Adding Other Repositories
There are some reasons for which you might want to add non-Ubuntu repositories to your list of
software sources. Caution: To avoid trouble with your sytem, only add repositories that are
trustworthy and that are known to work on Ubuntu systems!
•You can add custom software repositories by adding the apt repository line of your software
source to the end of thesources.list file. It should look something like this:
deb http://mirror3.ubuntulinux.nl/ hardy-seveas freenx
•Don't forget to make apt aware of your changes:
60. sudo apt-get update
Adding Launchpad PPA Repositories
Adding Launchpad PPA (Personal Package Archive) is possible conveniently via the
command: add-apt-repository. This command is similar to "addrepo" on Debian.
•The command updates your sources.list file or adds/edits files under sources.list.d/. Type man
add-apt-repository for detailed help.
•If a public key is required and available it is automatically downloaded and registered.
•Should be installed by default. On older or minimal Ubuntu releases, you may have to
install software-properties-common and/or python-software-properties first (sudo apt-get
install python-software-properties)
sudo add-apt-repository ppa:<repository-name>
Example: sudo add-apt-repository ppa:nhandler/ppa
Enabling Repositories with a (non-interactive) Script
This section seemed obsolete due to the add-apt-repository command, thus it has been
removed.
The Basics
Ubuntu uses apt for package management. Apt stores a list of repositories or software channels in
the file
/etc/apt/sources.list
and in any file with the suffix .list under the directory
/etc/apt/sources.list.d/
See man sources.list for more about this storage mechanism.
By editing these files from the command line, we can add, remove, or temporarily disable
software repositories.
• Note: It's always a good idea to backup a configuration file like sources.list before you edit
it. To do so, issue the following command:
sudo cp /etc/apt/sources.list /etc/apt/sources.list.backup
Typically, the beginning of the file /etc/apt/sources.list looks like this:
# sources.list
#deb cdrom:[Ubuntu 13.10 _Saucy Salamander_ - Release i386 (20131016.1)]/ saucy
main restricted
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://ch.archive.ubuntu.com/ubuntu/ saucy main restricted
deb-src http://ch.archive.ubuntu.com/ubuntu/ saucy main restricted
#...
61. Explanation of the Repository Format
•All the lines beginning with one or two hashes (#) are comments, for information only.
•The lines without hashes are apt repository lines. Here's what they say:
•deb: These repositories contain binaries or precompiled packages. These repositories are
required for most users.
•deb-src: These repositories contain the source code of the packages. Useful for developers.
•http://archive.ubuntu.com/ubuntu: The URI (Uniform Resource Identifier), in this case a
location on the internet. See the official mirror list or the self-maintained mirror list to find other
mirrors.•saucy is the release name or version of your distribution.
•main & restricted are the section names or components. There can be several section names,
separated by spaces.
Other Versions
IMPORTANT
For other Ubuntu releases you would replace the 'saucy' with the current version you
have installed ('precise', 'quantal', 'raring', 'saucy', 'trusty', ...) Type lsb_release -sc to
find out your release.
Since our release is precise
root@ubuntu:/home/arun# nano /etc/bash.bashrc
root@ubuntu:/home/arun# nano /etc/apt/sources.list
root@ubuntu:/home/arun# lsb_release -sc
precise
root@ubuntu:/home/arun#
deb http://us.archive.ubuntu.com/ubuntu/ precise universe
deb-src http://us.archive.ubuntu.com/ubuntu/ precise universe
deb http://us.archive.ubuntu.com/ubuntu/ precise-updates universe
deb-src http://us.archive.ubuntu.com/ubuntu/ precise-updates universe
sudo add-apt-repository "deb http://us.archive.ubuntu.com/ubuntu/ precise universe
multiverse"
sudo add-apt-repository "deb http://us.archive.ubuntu.com/ubuntu/ precise-updates
universe multiverse"
deb http://archive.canonical.com/ubuntu precise partner
deb-src http://archive.canonical.com/ubuntu precise partner
Depending on your location, you should replace 'us.' by another country code, referring to a
mirror server in your region. Check sources.list to see what is used!
Best mirror server for location can be find from software uptodate
63. newly some server to sources.list above commented with #newly added us server
Important to refresh otherwise wont reflect
Don't forget to retrieve the updated package lists:
•sudo apt-get update
64. You can add custom software repositories by adding the apt repository line of your software
source to the end of thesources.list file. It should look something like this:
deb http://mirror3.ubuntulinux.nl/ hardy-seveas freenx
•Don't forget to make apt aware of your changes:
sudo apt-get update
Adding Launchpad PPA Repositories
Adding Launchpad PPA (Personal Package Archive) is possible conveniently via the
command: add-apt-repository. This command is similar to "addrepo" on Debian.
•The command updates your sources.list file or adds/edits files under sources.list.d/. Type man
add-apt-repository for detailed help.
•If a public key is required and available it is automatically downloaded and registered.
•Should be installed by default. On older or minimal Ubuntu releases, you may have to
install software-properties-common and/or python-software-properties first (sudo apt-get
install python-software-properties)
sudo add-apt-repository ppa:<repository-name>
Example: sudo add-apt-repository ppa:nhandler/ppa