Windowshadoop

`
Install openssh and if available cmake , make

After installation run cygwin as Admin important other wise it won't
work

Arun@Arun-PC:ssh-host-config
click yes to all prompt questions and enter
Arun@Arun-PC:net start sshd

check whether 22 port is in use by below command
Arun@Arun-PC:net stat -a|grep 22

UserName Arun and passwd is case sensitive enter username same case
as enter in the user creation

From Apache Hadoop 2.2.0 release hadoop officially supports for running Hadoop on Microsoft
Windows as well. But the bin distribution ofApache Hadoop 2.2.0 release does not contain some
windows native components (like winutils.exe, hadoop.dll etc). As a result, if we try to run Hadoop
in windows, we'll encounter ERROR util.Shell: Failed to locate the winutils binary in the hadoop
binary path.
To fix this follow the steps:
STEP 1:INSTALL TOOLS TO RUN HADOOP ON WINDOWS
Install cygwin and during installation search openssh and check openssh checkbox and install.
once sshd is started port 22 is open and can be used by hadoop namenode for communicating with
the hdfs.

Secure Shell, or SSH, is a cryptographic (encrypted) network protocol to allow remote login and other network services to operate securely over
an unsecured network.[1]
SSH provides a secure channel over an unsecured network in a client-server architecture, connecting an SSH client application with an SSH
server.[2]
Common applications include remote command-line login and remote command execution, but any network service can be secured with
SSH. The protocol specification distinguishes between two major versions, referred to as SSH-1 and SSH-2.
The most visible application of the protocol is for access to shell accounts on Unix-like operating systems, but it sees some limited use
on Windows as well. In 2015, Microsoft announced that they would include native support for SSH in a future release.[3]
SSH was designed as a replacement for Telnet and for unsecured remote shell protocols such as the Berkeley rlogin, rsh, and rexec protocols.
Those protocols send information, notably passwords, in plaintext, rendering them susceptible to interception and disclosure using packet
analysis.[4]
The encryption used by SSH is intended to provide confidentiality and integrity of data over an unsecured network, such as the Internet,
although files leaked by Edward Snowden indicate that the National Security Agency can sometimes decrypt SSH, allowing them to read the
content of SSH sessions
SSH uses public-key cryptography to authenticate the remote computer and allow it to authenticate the user, if necessary.[2]
There are several
ways to use SSH; one is to use automatically generated public-private key pairs to simply encrypt a network connection, and then
use password authentication to log on.
Another is to use a manually generated public-private key pair to perform the authentication, allowing users or programs to log in without having to
specify a password. In this scenario, anyone can produce a matching pair of different keys (public and private). The public key is placed on all
computers that must allow access to the owner of the matching private key (the owner keeps the private key secret). While authentication is based
on the private key, the key itself is never transferred through the network during authentication. SSH only verifies whether the same person
offering the public key also owns the matching private key. In all versions of SSH it is important to verify unknown public keys, i.e. associate the
public keys with identities, before accepting them as valid. Accepting an attacker's public key without validation will authorize an unauthorized
attacker as a valid user.
SSH is typically used to log into a remote machine and execute commands, but it also supports tunneling, forwarding TCP
ports and X11 connections; it can transfer files using the associated SSH file transfer (SFTP) or secure copy (SCP) protocols.[2]
SSH uses
the client-server model.
The standard TCP port 22 has been assigned for contacting SSH servers.[7]
An SSH client program is typically used for establishing connections to an SSH daemon accepting remote connections. Both are commonly present on most
modern operating systems, including Mac OS X, most distributions of GNU/Linux, OpenBSD, FreeBSD, NetBSD, Solaris and OpenVMS.
Notably, Windows is one of the few modern desktop/server OSs that does not include SSH by default. Proprietary, freeware and open source (e.g. PuTTY,
[8]
and the version of OpenSSH which is part of Cygwin[9]
) versions of various levels of complexity and completeness exist. Native Linux file managers
(e.g. Konqueror) can use the FISH protocol to provide a split-pane GUI with drag-and-drop. The open source Windows program WinSCP[10]
provides similar
file management (synchronization, copy, remote delete) capability using PuTTY as a back-end. Both WinSCP[11]
and PuTTY[12]
are available packaged to run
directly off of a USB drive, without requiring installation on the client machine. Setting up an SSH server in Windows typically involves installation (e.g. via
installing Cygwin[13]
).
SSH is important in cloud computing to solve connectivity problems, avoiding the security issues of exposing a cloud-based virtual machine directly on the
Internet. An SSH tunnel can provide a secure path over the Internet, through a firewall to a virtual machine.[1

If above service stopped error occurs
If above service stopped error occur, when try to login to putty error occurs. Also make sure there is
no firefall or antivirus blocking.

Points to remember:
1. Install MS Windows SDK v7.v full with all components.
2. For me patch was required in one of the pom
3. Connect to internet
4. Follow this post step by step
Architecture
 The SSH-2 protocol has an internal architecture (defined in RFC 4251) with well-separated layers, namely:
 The transport layer (RFC 4253). This layer handles initial key exchange as well as server authentication, and sets up encryption, compression and integrity
verification. It exposes to the upper layer an interface for sending and receiving plaintext packets with sizes of up to 32,768 bytes each (more can be allowed by
the implementation). The transport layer also arranges for key re-exchange, usually after 1 GB of data has been transferred or after 1 hour has passed, whichever
occurs first.
 The user authentication layer (RFC 4252). This layer handles client authentication and provides a number of authentication methods. Authentication is client-
driven: when one is prompted for a password, it may be the SSH client prompting, not the server. The server merely responds to the client's authentication
requests. Widely used user-authentication methods include the following:
 password: a method for straightforward password authentication, including a facility allowing a password to be changed. Not all programs implement this
method.
 publickey: a method for public key-based authentication, usually supporting at least DSA or RSA keypairs, with other implementations also
supporting X.509certificates.
 keyboard-interactive (RFC 4256): a versatile method where the server sends one or more prompts to enter information and the client displays them and sends
back responses keyed-in by the user. Used to provide one-time password authentication such as S/Key or SecurID. Used by some OpenSSH configurations
when PAM is the underlying host-authentication provider to effectively provide password authentication, sometimes leading to inability to log in with a client
that supports just the plain password authentication method.
 GSSAPI authentication methods which provide an extensible scheme to perform SSH authentication using external mechanisms such as Kerberos
5 or NTLM, providing single sign-on capability to SSH sessions. These methods are usually implemented by commercial SSH implementations for use in
organizations, though OpenSSH does have a working GSSAPI implementation.
 The connection layer (RFC 4254). This layer defines the concept of channels, channel requests and global requests using which SSH services are provided. A
single SSH connection can host multiple channels simultaneously, each transferring data in both directions. Channel requests are used to relay out-of-band
channel-specific data, such as the changed size of a terminal window or the exit code of a server-side process. The SSH client requests a server-side port to be
forwarded using a global request. Standard channel types include:
 shell for terminal shells, SFTP and exec requests (including SCP transfers)
 direct-tcpip for client-to-server forwarded connections
 forwarded-tcpip for server-to-client forwarded connections
 The SSHFP DNS record (RFC 4255) provides the public host key fingerprints in order to aid in verifying the authenticity of the host.
 This open architecture provides considerable flexibility, allowing the use of SSH for a variety of purposes beyond a secure shell. The functionality of the
transport layer alone is comparable to Transport Layer Security(TLS); the user-authentication layer is highly extensible with custom authentication methods;
and the connection layer provides the ability to multiplex many secondary sessions into a single SSH connection, a feature comparable to BEEP and not
available in TLS.
Uses

Logging into OpenWrt via SSH usingPuTTY running on Windows.
SSH is a protocol that can be used for many applications across many platforms including most Unix variants (Linux, the BSDs including Apple's OS X,
and Solaris), as well as Microsoft Windows. Some of the applications below may require features that are only available or compatible with specific SSH
clients or servers. For example, using the SSH protocol to implement a VPN is possible, but presently only with the OpenSSH server and client
implementation.
Uses
SSH is a protocol that can be used for many applications across many platforms including most Unix variants (Linux, the BSDs including Apple's OS X,
and Solaris), as well as Microsoft Windows. Some of the applications below may require features that are only available or compatible with specific SSH
clients or servers. For example, using the SSH protocol to implement a VPN is possible, but presently only with the OpenSSH server and client
implementation.
1) For login to a shell on a remote host (replacing Telnet and rlogin)
2) For executing a single command on a remote host (replacing rsh)
3) For setting up automatic (passwordless) login to a remote server (for example, using OpenSSH[23]
)
4) Secure file transfer
5) In combination with rsync to back up, copy and mirror files efficiently and securely
6) For forwarding or tunneling a port (not to be confused with a VPN, which routes packets between different networks, or bridges two broadcast
domains into one).
7) For using as a full-fledged encrypted VPN. Note that only OpenSSH server and client supports this feature.
8) For forwarding X from a remote host (possible through multiple intermediate hosts)
9) For browsing the web through an encrypted proxy connection with SSH clients that support the SOCKS protocol.
10) For securely mounting a directory on a remote server as a filesystem on a local computer using SSHFS.
11) For automated remote monitoring and management of servers through one or more of the mechanisms discussed above.
12) For development on a mobile or embedded device that supports SSH.

Hadop common failed since need sdk in windows mvn package -Pdist,native-win -DskipTests –
Dtar or just mvn package -Pdist,native-win –DskipTests if any encoding error occurs with
Dtar. Not Dtar switch not used it just creates a folder inside hadoop-2.5.2-srchadoop-
disttarget hadoop-2.5.2 else hadoop-2.5.2.tar

[WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "p
rotoc": CreateProcess error=2, The system cannot find the file specified
[ERROR] stdout: []
install protoc buffer to environment variable and run.
Protocol buffers are a flexible, efficient, automated mechanism for serializing structured data – think XML, but smaller, faster, and
simpler. You define how you want your data to be structured once, then you can use special generated source code to easily
write and read your structured data to and from a variety of data streams and using a variety of languages. You can even update
your data structure without breaking deployed programs that are compiled against the "old" format.

need to run hadoop src with maven in Microsoft sdk if above error occurs.
To run in Microsoft sdk:

Select Start --> All Programs --> Microsoft Windows SDK v7.1 and open Windows SDK 7.1
Command Prompt. Change directory to Hadoop source code folder (c:hdfs). Execute mvn
package with options -Pdist,native-win -DskipTests -Dtar to create Windows binary tar
distribution.

issue is protoc buff is not set correctly in environment path:
After setting c:/protoc32/ in system environment variable path variable , below is output in window
ordinary command prompt.

STEP 2: GENERATE HADOOP FROM MAVEN SRC IN WINDOWS SDK
Run in windows sdk command prompt:

NOTE:
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (com
pile-ms-winutils) on project hadoop-common: Command execution failed. Cannot run
program "msbuild" (in directory "C:Hadoophadoop-2.5.2-srchadoop-common-proje
cthadoop-common"): CreateProcess error=2, The system cannot find the file speci
fied -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal o
rg.codehaus.mojo:exec-maven-plugin:1.2:exec (compile-ms-winutils) on project had
Environment variables:
%SystemRoot%;C:Program Files (x86)SkypePhone;C:Program
FilesJavajdk1.7.0_80bin;C:Mongo 2.6bin;C:WindowsSysWOW64;C:mule-standalone-
3.7.0bin;C:apache-maven-3.1.1bin;C:Program
FilesJavajdk1.7.0_80bin;C:cygwin64bin;C:cygwin64homeadminhadoop-
2.5.2bin;C:cygwin64homeadminhadoop-2.5.2sbin;C:Program FilesMicrosoft Windows
Performance Toolkit;C:protoc32;C:hadooputils;C:Program FilesMicrosoft
SDKsWindowsv7.1Bin;C:WindowsMicrosoft.NETFramework64v3.5
C:WindowsMicrosoft.NETFramework64v3.5 –MSBuild.exe

util.NativeCrc32"
[INFO]
[INFO] --- exec-maven-plugin:1.2:exec (compile-ms-winutils) @ hadoop-common ---
Build started 1/14/2016 10:43:27 PM.
Project "C:Hadoophadoop-2.5.2-srchadoop-common-projecthadoop-commonsrcmain
winutilswinutils.sln" on node 0 (default targets).
Building solution configuration "Release|Win32".
C:Hadoophadoop-2.5.2-srchadoop-common-projecthadoop-commonsrcmainwinutils
winutils.sln.cache(245,5): error MSB3411: Could not load the Visual C++ compone
nt "VCBuild.exe". If the component is not installed, either 1) install the Micro
soft Windows SDK for Windows Server 2008 and .NET Framework 3.5, or 2) install M
icrosoft Visual Studio 2008.
Done Building Project "C:Hadoophadoop-2.5.2-srchadoop-common-projecthadoop-c
ommonsrcmainwinutilswinutils.sln" (default targets) -- FAILED.
Build FAILED.
Run again by downloading vcbuild.exe manually from git but version is not compatible.
Build started 1/14/2016 11:05:33 PM.
Project "C:Hadoophadoop-2.5.2-srchadoop-common-projecthadoop-commonsrcmain
winutilswinutils.sln" on node 0 (default targets).
Building solution configuration "Release|Win32".
C:Hadoophadoop-2.5.2-srchadoop-common-projecthadoop-commonsrcmainwinutils
winutils.sln.cache(245,5): error MSB6003: The specified task executable "VCBuil
d.exe" could not be run. This version of %1 is not compatible with the version o
f Windows you're running. Check your computer's system information to see whethe
r you need a x86 (32-bit) or x64 (64-bit) version of the program, and then conta
ct the software publisher
Installed to fix this issue:
1. dotNetFx45_Full_setup standalone.
2. .net Framework 4.
3. VC-Compiler-KB2519277
NotFoundException
C:Hadoophadoop-2.5.2-src>mvn -X package -Pdist,native-win -DskipTests -Dtar
Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 20:52:2
2+0530)
Maven home: C:apache-maven-3.1.1bin..
Java version: 1.7.0_80, vendor: Oracle Corporation
Java home: C:Program FilesJavajdk1.7.0_80jre

Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
[INFO] Error stacktraces are turned on.
[DEBUG] Reading global settings from C:apache-maven-3.1.1bin..confsettings.
xml
[DEBUG] Reading user settings from C:Usersadmin.m2settings.xml
[DEBUG] Using local repository at C:Usersadmin.m2repository
[DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10.0 for C:U
sersadmin.m2repository
[INFO] Scanning for projects...
NOTE:
make sure encoding is UTF-8
if above error occurs install .net runtime framework 4.0 standalone and vc-compiler

error : You are attempting to build a Win32 application from an x64 environment. If using
theWindows 7.1 SDK build environment, type setenv /x86 [C:Hadoophadoop-2.5.2-srchadoop-
common-projecthadoop-commonsrcmainwinutilslibwinutils.vcxproj]

NOTE:
Microsoft does not have 64 bit of vc compiler hence to compile it in 64 follow
This steps:
1. Download and install the Windows Software Development Kit version 7.1. Visual C++ 2010
Express does not include a 64 bit compiler, but the SDK does. A link to the
SDK:http://msdn.microsoft.com/en-us/windowsserver/bb980924.aspx
2. Change your project configuration. Go to Properties of your project. On the top of the dialog
box there will be a "Configuration" drop-down menu. Make sure that selects "All Configurations."
There will also be a "Platform" drop-down that will read "Win32." Finally on the right there is a
"Configuration Manager" button - press it. In the dialog that comes up, find your project, hit the
Platform drop-down, select New, then select x64. Now change the "Active solution platform" drop-
down menu to "x64." When you return to the Properties dialog box, the "Platform" drop-down
should now read "x64."
3. Finally, change your toolset. In the Properties menu of your project, under Configuration
Properties | General, change Platform Toolset from "v100" to "Windows7.1SDK".
These steps have worked for me, anyway
Change win32 to x64 as mentioned above since vc++ editor is not there just change in the below
project configuration file.
C:Hadoophadoop-2.5.2-srchadoop-common-projecthadoop-commonsrcmainwinutilswinutils.sln
AS MENTIONED in the above desc uncomment all 32 Active solution platform.
Change win32 to x64 (find and replaced)
NOTE:
As like below it wont accept comment (i.e #) in global section(preSolution)
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Win32 = Debug|Win32
Debug|x64 = Debug|x64
Release|Win32 = Release|Win32
Release|x64 = Release|x64
EndGlobalSection

Installed vcbuild dir:
C:Program Files (x86)MSBuildMicrosoft.Cppv4.0PlatformsWin32PlatformToolsetsWindows7.1SDK
Error : You are attempting to build a Win32 application from an x64 environment. If using
theWindows 7.1 SDK build environment, type setenv /x86 [C:Hadoophadoop-2.5.2-srchadoop-common-
projecthadoop-commonsrcmainwinutilslibwinutils.vcxproj]
Above error coming from Microsoft.Cpp.Win32.Windows7.1SDK.targets file.
C:Program Files (x86)MSBuildMicrosoft.Cppv4.0PlatformsWin32PlatformToolsetsWindows7.1SDK Microsoft.Cpp.Win32.Windows7.1SDK.targets

Above Condition:"'$(WindowsSDKVersionOverride)' != ''and '$(TARGET_CPU)' == 'x64'" causes this
error . hence set the Environment like below.
Environment variable to set:
set WindowsSdkDir=C:Program FilesMicrosoft SDKsWindowsv7.1
set PATH=%WindowsSdkDir%Bin;%PATH%
set LIB=%WindowsSdkDir%Lib;%LIB%
set INCLUDE=%WindowsSdkDir%Include;%INCLUDE%
set TARGET_CPU=x64
After this all error gone but below error:
"C:Hadoophadoop-2.5.2-srchadoop-common-projecthadoop-commonsrcmainnative
native.sln" (default target) (1) ->
native.vcxproj" (default target) (2) ->
(GetReferenceAssemblyPaths target) ->
C:WindowsMicrosoft.NETFramework64v4.0.30319Microsoft.Common.Targets(847,9
): warning MSB3644: The reference assemblies for framework ".NETFramework,Versio
n=v4.0" were not found. To resolve this, install the SDK or Targeting Pack for t
his framework version or retarget your application to a version of the framework
for which you have the SDK or Targeting Pack installed. Note that assemblies wi
ll be resolved from the Global Assembly Cache (GAC) and will be used in place of
reference assemblies. Therefore your assembly may not be correctly targeted for
the framework you intend. [C:Hadoophadoop-2.5.2-srchadoop-common-projecthad
oop-commonsrcmainnativenative.vcxproj]
native.sln" (default target) (1) ->
native.vcxproj" (default target) (2) ->
(ClCompile target) ->
C:Program Files (x86)Microsoft Visual Studio 10.0VCincludeintrin.h(26): f
atal error C1083: Cannot open include file: 'ammintrin.h': No such file or direc
tory [C:Hadoophadoop-2.5.2-srchadoop-common-projecthadoop-commonsrcmainna
tivenative.vcxproj]
C:Program Files (x86)Microsoft Visual Studio 10.0VCincludeintrin.h(26): f
atal error C1083: Cannot open include file: 'ammintrin.h': No such file or direc
tory [C:Hadoophadoop-2.5.2-srchadoop-common-projecthadoop-commonsrcmainna
tivenative.vcxproj]
1 Warning(s)
2 Error(s)

to fix the above error fix is available in hadoop
I can not compile my C++ projects after installing ‘Security Update for Microsoft Visual Studio 2010 (KB2455033)’, see my yesterday’s post:
Security Update for Microsoft Visual Studio 2010 (KB2455033) failed with Code 643.
Visual Studio 2010 shows the following error:
1>------ Build started: Project: some_project_name_i_am_hiding, Configuration: Debug Win32 ------
1> stdafx.cpp
1>c:program files (x86)microsoft visual studio 10.0vcincludeintrin.h(26): fatal error C1083: Cannot open include file: 'ammintrin.h': No such file or directory
========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========
Error is caused because file intrin.h have reference to ammintrin.h, and after this update there is no such file. Relevant fragment from intrin.h:
#if defined(_M_IX86) || defined(_M_X64)
#include <immintrin.h>
#include <ammintrin.h>
i have reinstalled Visual Studio since : Windows Update KB2455033 breaks build with missing ammintrin.h.
see post by visual studio
https://connect.microsoft.com/VisualStudio/feedback/details/660584/windows-update-kb2455033-breaks-build-with-missing-ammintrin-h
Windows Update KB2455033 breaks build with missing ammintrin.h - by
The workaround is to install Visual Studio 2010 SP1. Download it form Microsoft Website: Microsoft Visual Studio 2010 Service Pack 1 (Installer).
File to install:
VS10sp1-KB983509.exe
Or just download ammintrin.m from
http://in.mathworks.com/matlabcentral/answers/uploaded_files/735/ammintrin.m
After this still getting below error obj linker error

to fix this when tried with setenv /x86 it clearly says try x64

hence try with setenv /x64 itself to fix linker error
Change win32 to x64 (find and replaced) as in winutils.sln, winutils.vcxproj, libwinutils.vcxproj
run again

NOTE:
In native.sln comment even mixed mode too. It shows native obj linker error previously above file.
finally issue fixed – working !!!.

see hadoop-2.5.2 gets generated.

STEPS:
1) download hadoop-src from the apache.
2) Download Microsoft .net Runtime 4.0 and VC-compiler.exe
3) change native,winutils folder inside hadoop-2.5.2-srchadoop-common-projecthadoop-
commonsrcmain to match the 64 bit version.changed version available under:
download the https://github.com/arunsadhasivam/hadoop/edit/master/README.md
4) download ammintrin.m from below url.
http://in.mathworks.com/matlabcentral/answers/uploaded_files/735/ammintrin.m
else update with latest version of VC-compiler.exe
The workaround is to install Visual Studio 2010 SP1. Download it form Microsoft Website: Microsoft Visual Studio 2010 Service
Pack 1 (Installer).
5) check whether the below environment variable is there
SDKsWindowsv7.1Bin;C:WindowsMicrosoft.NETFramework64v3.5

6) install windows sdk 1.7 version and make sure that it open without any error like below

7) set the below environment variable and then run
All are very important especially setenv else it uses /x86 and shows “trying to run win32 in 64”
setenv /x64
set TARGET_CPU=x64
it open new windows then enter all others

STEP 3: RUN MAVEN TO GENERATE NATIVE LIBRARY
After successful run , hadoop dist did not have lib folder , since lib folder is important for
running hadoop in windows .
when run with out native lib it shows below error. Since windows need native libraries

TO get the native libraries use the below command
mvn package -Pdist,native –DskipTests

download CMake
since hadoop requires this file to compile header file

Requirements:
1. Windows System
2. JDK 1.6+
3. Maven 3.0 or later
4. Findbugs 1.3.9 (if running findbugs)
5. ProtocolBuffer 2.5.0
6. CMake 2.6 or newer
7. Make
8. Windows SDK or Visual Studio 2010 Professional
9. Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip
10. zlib headers (if building native code bindings for zlib)
11. Internet connection for first build (to fetch all Maven and Hadoop dependencies)
If using Visual Studio, it must be Visual Studio 2010 Professional (not 2012). Do not use Visual
Studio Express. It does not support compiling for 64-bit, which is problematic if running a 64-bit
system. The Windows SDK is free to download here:
http://www.microsoft.com/en-us/download/details.aspx?id=8279
Building:
Keep the source code tree in a short path to avoid running into problems related to Windows
maximum path length limitation. (For example, C:hdc).
Run builds from a Windows SDK Command Prompt. (Start, All Programs, Microsoft Windows
SDK v7.1, Windows SDK 7.1 Command Prompt.)

JAVA_HOME must be set, and the path must not contain spaces. If the full path would contain
spaces, then use the Windows short path instead.
You must set the Platform environment variable to either x64 or Win32 depending on whether
you're running a 64-bit or 32-bit system. Note that this is case-sensitive. It must be "Platform",
not "PLATFORM" or "platform". Environment variables on Windows are usually case-
insensitive, but Maven treats them as case-sensitive. Failure to set this environment variable
correctly will cause msbuild to fail while building the native code in hadoop-common.
set Platform=x64 (when building on a 64-bit system) set Platform=Win32 (when building on a
32-bit system)
Several tests require that the user must have the Create Symbolic Links privilege.
All Maven goals are the same as described above with the exception that native code is built by
enabling the 'native-win' Maven profile. -Pnative-win is enabled by default when building on
Windows since the native components are required (not optional) on Windows.
If native code bindings for zlib are required, then the zlib headers must be
deployed on the build machine. Set the ZLIB_HOME environment variable to the directory
containing the headers.
set ZLIB_HOME=C:zlib-1.2.7
At runtime, zlib1.dll must be accessible on the PATH. Hadoop has been tested with zlib 1.2.7,
built using Visual Studio 2010 out of contribvstudiovc10 in
the zlib 1.2.7 source tree.
Building distributions:
 Build distribution with native code : mvn package [-Pdist][-Pdocs][-Psrc][-Dtar]

see now it looks to add native lib to the dist folder with native lib.
mingw-w64 –Make file to compile c ,c++ header filer

The mingw-w64 project is a complete runtime environment for gcc to support binaries native to Windows 64-bit and
32-bit operating systems.
CMAKE VS MAKE
Programmers have been using CMake and Make for a long time now. When you join a big company or start working
on a project with a large codebase, there are all these builds that you need to take care of. You must have seen
those “CMakeLists.txt” files floating around. You are supposed to run “cmake” and “make” commands on the
terminal. A lot of people just follow the instructions blindly, not really caring about why we need to do things in a
certain way. What is this whole build process and why is it structured this way? What are the differences between
CMake and Make? Does it matter? Are they interchangeable?
As it turns out, they are quite different. It is important to understand the differences between them to make sure you
don’t get yourself in trouble. Before getting into the differences, let’s first see what they are.
Make
The way in which we design a software system is that we first write code, then the compiler compiles it and creates
executable files. These executable files are the ones that carry out the actual task. “Make” is a tool that controls the
generation of executables and other non-source files of a program from the program’s source files.
The “Make” tool needs to know how to build your program. It gets its knowledge of how to build your program from a
file called the “makefile”. This makefile lists each of the non-source files and how to compute it from other files. When
you write a program, you should write a makefile for it, so that it is possible to use “Make” to build and install the
program. Simple stuff! If you didn’t understand it, go back and read the paragraph again because it’s important for the
next part.
Why do we need “Make”?
The reason we need “Make” is because it enables the end user to build and install your package without knowing the
details of how it’s done. Every project comes with its own rules and nuances, and it can get quite painful every time
you have a new collaborator. That’s the reason we have this makefile. The details of the build process are actually
recorded in the makefile that you supply. “Make” figures out automatically which files it needs to update, based on
which source files have changed. It also automatically determines the proper order for updating the files, in case one
non-source file depends on another non-source file.

Recompiling the entire program every time we change a small part of the system would be inefficient. Hence, if you
change a few source files and then run “Make”, it doesn’t recompile the whole thing. It updates only those non-source
files that depend directly or indirectly on the source files that you changed. Pretty neat! “Make” is not limited to any
particular language. For each non-source file in the program, the makefile specifies the shell commands to compute
it. These shell commands can run a compiler to produce an object file, the linker to produce an executable, ar to
update a library, Makeinfo to format documentation, etc. “Make” is not limited to just building a package either. You
can also use “Make” to control installing or uninstalling a package, generate tags tables for it, or anything else you
want to do often enough to make it worth while writing down how to do it.
CMake
CMake stands for Cross-platform Make. CMake recognizes which compilers to use for a given kind of source. In case
you didn’t know, you can’t use the same compiler to build all the different kinds of sources. You can do this manually
every time you want to build your project, but it would be tedious and painful. CMake invokes the right sequence of
commands for each type of target. Therefore, there is no explicit specification of commands like $(CC).
For coding junkies who really want the gory details, read on. If you are not into all that, you can skip to the next
section. All the usual compiler/linker flags dealing with the inclusion of header files, libraries, etc are replaced by
platform independent and build system independent commands. Debugging flags are included by either setting the
variable CMAKE_BUILD_TYPE to “Debug”, or by passing it to CMake when invoking the program:
cmake -DCMAKE_BUILD_TYPE:STRING=Debug.
CMake also offers the platform independent inclusion of the ‘-fPIC’ flag (via the POSITION_INDEPENDENT_CODE
property) and many others. Still, more obscure settings can be implemented by hand in CMake just as well as in a
Makefile (by using COMPILE_FLAGS and similar properties). Of course CMake really starts to shine when third party
libraries (like OpenGL) are included in a portable manner.
What is the difference?
The build process has one step if you use a Makefile, namely typing “make” at the command line. For CMake, there
are two steps: First, you need to setup your build environment (either by typing cmake <source_dir> in your build
directory or by running some GUI client). This creates a makefile or something equivalent, depending on the build
system of your choice (e.g. Make on *nix, VC++ or MinGW on Windows, etc). The build system can be passed to
CMake as a parameter. However, CMake makes reasonable default choices depending on your system
configuration. Second, you perform the actual build in the selected build system.
We are going to jump into the GNU build system territory here. If you are not familiar with that, this paragraph might
look like jibber-jabber to you. Alright, now that I have given the statutory warning, let’s move on! We can compare
CMake with Autotools. When we do that, we can see the shortcomings of Make, and they form the reason for the
creation of Autotools. We can also see the obvious advantages of CMake over Make. Autoconf solves an important
problem i.e. reliable discovery of system-specific build and runtime information. But this is only a small part in the
development of portable software. To this end, the GNU project has developed a suite of integrated utilities to finish
the job Autoconf started: the GNU build system, whose most important components are Autoconf, Automake, and
Libtool.
“Make” can’t do that, at least not without modifying it anyway! You can make it do all that stuff but it would take a lot
of time maintaining it across platforms. CMake solves the same problem, but at the same time, it has a few
advantages over the GNU Build System:
 The language used to write CMakeLists.txt files is readable and easier to understand.
 It doesn’t only rely on “Make” to build the project.
 It supports multiple generators like Xcode, Eclipse, Visual Studio, etc.
When comparing CMake with Make, there are several advantages of using CMake:

 Cross platform discovery of system libraries.
 Automatic discovery and configuration of the toolchain.
 Easier to compile your files into a shared library in a platform agnostic way, and in general easier to use than make.
CMake does more than just “make”, so it can be more complex. In the long run, it’s better to learn how to use it. If you
have just a small project on only one platform, then maybe “Make” can do a better job.

STEPS FOR INSTALLING HADOOP ONWINDOWS
Steps to generate hadoop executable and native files
1. JDK 1.6+
2. Maven 3.0 or later
3. Findbugs 1.3.9 (if running findbugs)
4. ProtocolBuffer 2.5.0 - marshalling
5. CMake(cross platform make) 2.6 or newer - give instructions commands to be
compiled by make
6. Make - to build executables by running the cmake scripts.
7. Windows SDK or Visual Studio 2010 Professional
8. Unix command-line tools from GnuWin32 or Cygwin: sh, mkdir, rm, cp, tar, gzip – to
connect to port 22 since hadoop name node listen to port 22.
9. zlib headers (if building native code bindings for zlib)- make sure zlib1.dll on class
path.
10. Internet connection for first build (to fetch all Maven and Hadoop dependencies)
11. Make sure try you set all the below environment variable before run maven or atleast
setenv /x64 . since setenv is very important without which it wont work.
setenv /x64
set TARGET_CPU=x64
12. make sure set hadooputils - C:hadooputils can be downloaded from
https://github.com/arunsadhasivam/hadoop/tree/master/hadooputils%20-complete
13. download hadood src and change hadood-common project (native,winutils) folder in
C:Hadoophadoop-2.5.2-srchadoop-common-projecthadoop-commonsrcmain with
corrected files according to 64 bit compilation from github.
https://github.com/arunsadhasivam/hadoop/hadoopWindows .
14. run maven with below command to compile src initially.
mvn package -Pdist,native-win –DskipTests
15. next run to get native file separately for that need CMake and Make exe in classpath.
mvn package -Pdist,native –DskipTest
NOTE:
Software needed to be installed in order are:
1.dotNetFx40_Full_x86_x64.exe
2.winsdk_web.exe
3.VC-Compiler-KB251927.exe
4.cmake-3.3.2-win32-x86.exe
5.mingw-w64-install.exe after install with 64 format rename mingw-make to make

Environment variable:
SDKsWindowsv7.1Bin;C:WindowsMicrosoft.NETFramework64v3.5;C:Program Files
(x86)Gitbin;C:Program Files (x86)CMakebin;C:zlib128
C:hadooputils –some exe like winutils.exe need to be downloaded manually
Also better to have git.exe and tortoise svn.exe in environment variable path since it shows
warning.

mingw-w64:
NOTE:
Renamed ming64-make to make since maven needs executable make.
make sure it points to only one version of .net framework. See it refer 4.0 but only 3.5 is in
environment PATH system variable.

As you can see older version does not impact only the latest version causes the problem. Since
like in below it take precedence.
Although .net 3.5 is needed to run by Microsoft sdk 7.1 below but all above v4.0.30319 gets
installed during .net runtime 4.0 installtion.which is needed.removing that show error during
compilation. So don’t remove v4.0.30319 . see below screen shot showing error.

since windows sdk uses below batch you can see it use .net 3.5 only. Since above I installed
C:WindowsSystem32cmd.exe /E:ON /V:ON /T:0E /K "C:Program FilesMicrosoft
SDKsWindowsv7.1BinSetEnv.cmd"

others not needed in redistributable pacakage

Environment Variable:
FilesJavajdk1.7.0_80bin;C:cygwin64bin;C:cygwin64homehadoop-
2.2.0bin;C:cygwin64homehadoop-2.2.0sbin;C:protoc32;C:Program FilesMicrosoft
SDKsWindowsv7.1Bin;C:WindowsMicrosoft.NETFramework64v3.5;C:Program Files
(x86)Gitbin;
works fine with hadoop-2.2.0 and native files gets generated

in hadoop-2.5.2 some files native files are missing
NOTE:
important to note that bin folder should contains .dll and .exe file (winutils.exe
for hadoop 2.2) which gets generated only for hadoop 2.2

[WARNING] C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainjavaorgapachehadoopsecuritySecurityUtil.java:[596,20]
IPAddressUtil is internal proprietary API and may be removed in a future release
[INFO]
[INFO] --- native-maven-plugin:1.0-alpha-7:javah (default) @ hadoop-common ---
[INFO] cmd.exe /X /C ""C:Program FilesJavajdk1.7.0_80binjavah" -d C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-
commontargetnativejavah -classpath "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-
commontargetclasses;C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-annotationstargethadoop-annotations-
2.2.0.jar;C:Program FilesJavajdk1.7.0_80jre..libtools.jar;C:Usersadmin.m2repositorycomgoogleguavaguava11.0.2guava-
11.0.2.jar;C:Usersadmin.m2repositorycomgooglecodefindbugsjsr3051.3.9jsr305-1.3.9.jar;C:Usersadmin.m2repositorycommons-
clicommons-cli1.2commons-cli-1.2.jar;C:Usersadmin.m2repositoryorgapachecommonscommons-math2.1commons-math-
2.1.jar;C:Usersadmin.m2repositoryxmlencxmlenc0.52xmlenc-0.52.jar;C:Usersadmin.m2repositorycommons-httpclientcommons-
httpclient3.1commons-httpclient-3.1.jar;C:Usersadmin.m2repositorycommons-codeccommons-codec1.4commons-codec-
1.4.jar;C:Usersadmin.m2repositorycommons-iocommons-io2.1commons-io-2.1.jar;C:Usersadmin.m2repositorycommons-netcommons-
net3.1commons-net-3.1.jar;C:Usersadmin.m2repositoryjavaxservletservlet-api2.5servlet-api-
2.5.jar;C:Usersadmin.m2repositoryorgmortbayjettyjetty6.1.26jetty-6.1.26.jar;C:Usersadmin.m2repositoryorgmortbayjettyjetty-
util6.1.26jetty-util-6.1.26.jar;C:Usersadmin.m2repositorycomsunjerseyjersey-core1.9jersey-core-
1.9.jar;C:Usersadmin.m2repositorycomsunjerseyjersey-json1.9jersey-json-
1.9.jar;C:Usersadmin.m2repositoryorgcodehausjettisonjettison1.1jettison-1.1.jar;C:Usersadmin.m2repositorystaxstax-api1.0.1stax-
api-1.0.1.jar;C:Usersadmin.m2repositorycomsunxmlbindjaxb-impl2.2.3-1jaxb-impl-2.2.3-
1.jar;C:Usersadmin.m2repositoryjavaxxmlbindjaxb-api2.2.2jaxb-api-
2.2.2.jar;C:Usersadmin.m2repositoryjavaxactivationactivation1.1activation-
1.1.jar;C:Usersadmin.m2repositoryorgcodehausjacksonjackson-jaxrs1.8.8jackson-jaxrs-
1.8.8.jar;C:Usersadmin.m2repositoryorgcodehausjacksonjackson-xc1.8.8jackson-xc-
1.8.8.jar;C:Usersadmin.m2repositorycomsunjerseyjersey-server1.9jersey-server-1.9.jar;C:Usersadmin.m2repositoryasmasm3.2asm-
3.2.jar;C:Usersadmin.m2repositorycommons-loggingcommons-logging1.1.1commons-logging-
1.1.1.jar;C:Usersadmin.m2repositorylog4jlog4j1.2.17log4j-1.2.17.jar;C:Usersadmin.m2repositorynetjavadevjets3tjets3t0.6.1jets3t-
0.6.1.jar;C:Usersadmin.m2repositorycommons-langcommons-lang2.5commons-lang-2.5.jar;C:Usersadmin.m2repositorycommons-
configurationcommons-configuration1.6commons-configuration-1.6.jar;C:Usersadmin.m2repositorycommons-collectionscommons-
collections3.2.1commons-collections-3.2.1.jar;C:Usersadmin.m2repositorycommons-digestercommons-digester1.8commons-digester-
1.8.jar;C:Usersadmin.m2repositorycommons-beanutilscommons-beanutils1.7.0commons-beanutils-
1.7.0.jar;C:Usersadmin.m2repositorycommons-beanutilscommons-beanutils-core1.8.0commons-beanutils-core-
1.8.0.jar;C:Usersadmin.m2repositoryorgslf4jslf4j-api1.7.5slf4j-api-1.7.5.jar;C:Usersadmin.m2repositoryorgcodehausjacksonjackson-
core-asl1.8.8jackson-core-asl-1.8.8.jar;C:Usersadmin.m2repositoryorgcodehausjacksonjackson-mapper-asl1.8.8jackson-mapper-asl-
1.8.8.jar;C:Usersadmin.m2repositoryorgapacheavroavro1.7.4avro-
1.7.4.jar;C:Usersadmin.m2repositorycomthoughtworksparanamerparanamer2.3paranamer-
2.3.jar;C:Usersadmin.m2repositoryorgxerialsnappysnappy-java1.0.4.1snappy-java-
1.0.4.1.jar;C:Usersadmin.m2repositorycomgoogleprotobufprotobuf-java2.5.0protobuf-java-2.5.0.jar;C:Hadoophadoop-2.2.0-srchadoop-
common-projecthadoop-authtargethadoop-auth-2.2.0.jar;C:Usersadmin.m2repositorycomjcraftjsch0.1.42jsch-
0.1.42.jar;C:Usersadmin.m2repositoryorgapachezookeeperzookeeper3.4.5zookeeper-
3.4.5.jar;C:Usersadmin.m2repositoryorgapachecommonscommons-compress1.4.1commons-compress-
1.4.1.jar;C:Usersadmin.m2repositoryorgtukaanixz1.0xz-1.0.jar" org.apache.hadoop.io.compress.zlib.ZlibCompressor
org.apache.hadoop.io.compress.zlib.ZlibDecompressor org.apache.hadoop.security.JniBasedUnixGroupsMapping
org.apache.hadoop.io.nativeio.NativeIO org.apache.hadoop.security.JniBasedUnixGroupsNetgroupMapping
org.apache.hadoop.io.compress.snappy.SnappyCompressor org.apache.hadoop.io.compress.snappy.SnappyDecompressor
org.apache.hadoop.io.compress.lz4.Lz4Compressor org.apache.hadoop.io.compress.lz4.Lz4Decompressor org.apache.hadoop.util.NativeCrc32"
[INFO]
[INFO] --- exec-maven-plugin:1.2:exec (compile-ms-winutils) @ hadoop-common ---
Build started 2/13/2016 6:55:01 PM.
Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilswinutils.sln" on node 1 (default targets).
ValidateSolutionConfiguration:
Building solution configuration "Release|x64".
Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilswinutils.sln" (1) is building
"C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilswinutils.vcxproj.metaproj" (2) on node 1 (default
targets).
Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilswinutils.vcxproj.metaproj" (2) is building
"C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilslibwinutils.vcxproj" (3) on node 1 (default targets).
PrepareForBuild:
Creating directory "......targetwinutilsx64Release".
Creating directory "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontarget/bin/".
InitializeBuildStatus:
Creating "......targetwinutilsx64Releaselibwinutils.unsuccessfulbuild" because "AlwaysCreate" was specified.
ClCompile:

C:Program Files (x86)Microsoft Visual Studio 10.0VCbinamd64CL.exe /c /Zi /nologo /W3 /WX- /O2 /Oi /GL /D WIN32 /D NDEBUG /D
_CONSOLE /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Fo"......targetwinutilsx64Release"
/Fd"......targetwinutilsx64ReleasevcWindows7.1SDK.pdb" /Gd /TC /errorReport:queue libwinutils.c
libwinutils.c
Lib:
C:Program Files (x86)Microsoft Visual Studio 10.0VCbinamd64Lib.exe /OUT:"C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-
commontarget/bin/libwinutils.lib" /NOLOGO /LTCG ......targetwinutilsx64Releaselibwinutils.obj
libwinutils.vcxproj -> C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontarget/bin/libwinutils.lib
FinalizeBuildStatus:
Deleting file "......targetwinutilsx64Releaselibwinutils.unsuccessfulbuild".
Touching "......targetwinutilsx64Releaselibwinutils.lastbuildstate".
Done Building Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilslibwinutils.vcxproj" (default targets).
Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilswinutils.vcxproj.metaproj" (2) is building
"C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilswinutils.vcxproj" (4) on node 1 (default targets).
Creating "......targetwinutilsx64Releasewinutils.unsuccessfulbuild" because "AlwaysCreate" was specified.
ClCompile:
C:Program Files (x86)Microsoft Visual Studio 10.0VCbinamd64CL.exe /c /Zi /nologo /W3 /WX- /O2 /Oi /GL /D WIN32 /D NDEBUG /D
_CONSOLE /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /Gy /fp:precise /Zc:wchar_t /Zc:forScope /Fo"......targetwinutilsx64Release"
/Fd"......targetwinutilsx64ReleasevcWindows7.1SDK.pdb" /Gd /TC /errorReport:queue readlink.c symlink.c systeminfo.c chmod.c chown.c groups.c
hardlink.c task.c ls.c main.c
readlink.c
symlink.c
systeminfo.c
chmod.c
chown.c
groups.c
hardlink.c
task.c
ls.c
main.c
Link:
C:Program Files (x86)Microsoft Visual Studio 10.0VCbinamd64link.exe /ERRORREPORT:QUEUE /OUT:"C:Hadoophadoop-2.2.0-srchadoop-
common-projecthadoop-commontarget/bin/winutils.exe" /INCREMENTAL:NO /NOLOGO kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib
advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib odbccp32.lib /MANIFEST
/ManifestFile:"......targetwinutilsx64Releasewinutils.exe.intermediate.manifest" /MANIFESTUAC:"level='asInvoker' uiAccess='false'" /DEBUG
/PDB:"C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontargetbinwinutils.pdb" /SUBSYSTEM:CONSOLE /OPT:REF
/OPT:ICF /LTCG /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-
commontarget/bin/winutils.lib" /MACHINE:X64 ......targetwinutilsx64Releasereadlink.obj
......targetwinutilsx64Releasesymlink.obj
......targetwinutilsx64Releasesysteminfo.obj
......targetwinutilsx64Releasechmod.obj
......targetwinutilsx64Releasechown.obj
......targetwinutilsx64Releasegroups.obj
......targetwinutilsx64Releasehardlink.obj
......targetwinutilsx64Releasetask.obj
......targetwinutilsx64Releasels.obj
......targetwinutilsx64Releasemain.obj
"C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontargetbinlibwinutils.lib"
Generating code

Finished generating code
winutils.vcxproj -> C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontarget/bin/winutils.exe
Manifest:
C:Program FilesMicrosoft SDKsWindowsv7.1binx64mt.exe /nologo /verbose /outputresource:"C:Hadoophadoop-2.2.0-srchadoop-common-
projecthadoop-commontarget/bin/winutils.exe;#1" /manifest ......targetwinutilsx64Releasewinutils.exe.intermediate.manifest
Deleting file "......targetwinutilsx64Releasewinutils.unsuccessfulbuild".
Touching "......targetwinutilsx64Releasewinutils.lastbuildstate".
Done Building Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilswinutils.vcxproj" (default targets).
Done Building Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilswinutils.vcxproj.metaproj" (default
targets).
Done Building Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainwinutilswinutils.sln" (default targets).
Build succeeded.
0 Warning(s)
0 Error(s)
Time Elapsed 00:00:13.96
[INFO]
[INFO] --- exec-maven-plugin:1.2:exec (compile-ms-native-dll) @ hadoop-common ---
Build started 2/13/2016 6:55:16 PM.
Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.sln" on node 1 (default targets).
ValidateSolutionConfiguration:
Building solution configuration "Release|x64".
Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.sln" (1) is building "C:Hadoophadoop-2.2.0-
srchadoop-common-projecthadoop-commonsrcmainnativenative.vcxproj" (2) on node 1 (default targets).
PrepareForBuild:
Creating directory "......targetnativeRelease".
Creating "......targetnativeReleasenative.unsuccessfulbuild" because "AlwaysCreate" was specified.
ClCompile:
C:Program Files (x86)Microsoft Visual Studio 10.0VCbinamd64CL.exe /c /I..winutilsinclude /I......targetnativejavah /I"C:Program
FilesJavajdk1.7.0_80include" /I"C:Program FilesJavajdk1.7.0_80includewin32" /I.src /Zi /nologo /W3 /WX- /O2 /Oi /GL /D WIN32 /D
NDEBUG /D _WINDOWS /D _USRDLL /D NATIVE_EXPORTS /D _WINDLL /D _UNICODE /D UNICODE /Gm- /EHsc /MD /GS /Gy /fp:precise
/Zc:wchar_t /Zc:forScope /Fo"......targetnativeRelease" /Fd"......targetnativeReleasevcWindows7.1SDK.pdb" /Gd /TC /wd4244 /errorReport:queue
srcorgapachehadoopiocompresslz4lz4.c srcorgapachehadoopiocompresslz4Lz4Compressor.c
srcorgapachehadoopiocompresslz4Lz4Decompressor.c srcorgapachehadoopionativeiofile_descriptor.c
srcorgapachehadoopionativeioNativeIO.c srcorgapachehadoopsecurityJniBasedUnixGroupsMappingWin.c srcorgapachehadooputilbulk_crc32.c
srcorgapachehadooputilNativeCodeLoader.c srcorgapachehadooputilNativeCrc32.c
lz4.c
Lz4Compressor.c
Lz4Decompressor.c
file_descriptor.c
NativeIO.c
JniBasedUnixGroupsMappingWin.c
srcorgapachehadoopsecurityJniBasedUnixGroupsMappingWin.c(48): warning C4047: 'function' : 'LPSTR' differs in levels of indirection from 'LPSTR
*' [C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.vcxproj]
srcorgapachehadoopsecurityJniBasedUnixGroupsMappingWin.c(48): warning C4024: 'FormatMessageA' : different types for formal and actual
parameter 5 [C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.vcxproj]

bulk_crc32.c
NativeCodeLoader.c
NativeCrc32.c
Link:
C:Program Files (x86)Microsoft Visual Studio 10.0VCbinamd64link.exe /ERRORREPORT:QUEUE /OUT:"C:Hadoophadoop-2.2.0-
srchadoop-common-projecthadoop-commontarget/bin/hadoop.dll" /INCREMENTAL:NO /NOLOGO /LIBPATH:......targetbin Ws2_32.lib
libwinutils.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib uuid.lib odbc32.lib
odbccp32.lib /MANIFEST /ManifestFile:"......targetnativeReleasehadoop.dll.intermediate.manifest" /MANIFESTUAC:"level='asInvoker'
uiAccess='false'" /DEBUG /PDB:"C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontargetbinhadoop.pdb"
/SUBSYSTEM:WINDOWS /OPT:REF /OPT:ICF /LTCG /TLBID:1 /DYNAMICBASE /NXCOMPAT /IMPLIB:"C:Hadoophadoop-2.2.0-
srchadoop-common-projecthadoop-commontarget/bin/hadoop.lib" /MACHINE:X64 /DLL ......targetnativeReleaselz4.obj
......targetnativeReleaseLz4Compressor.obj
......targetnativeReleaseLz4Decompressor.obj
......targetnativeReleasefile_descriptor.obj
......targetnativeReleaseNativeIO.obj
......targetnativeReleaseJniBasedUnixGroupsMappingWin.obj
......targetnativeReleasebulk_crc32.obj
......targetnativeReleaseNativeCodeLoader.obj
......targetnativeReleaseNativeCrc32.obj
Creating library C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontarget/bin/hadoop.lib and object C:Hadoophadoop-2.2.0-
srchadoop-common-projecthadoop-commontarget/bin/hadoop.exp
Generating code
Finished generating code
native.vcxproj -> C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontarget/bin/hadoop.dll
Manifest:
C:Program FilesMicrosoft SDKsWindowsv7.1binx64mt.exe /nologo /verbose /outputresource:"C:Hadoophadoop-2.2.0-srchadoop-common-
projecthadoop-commontarget/bin/hadoop.dll;#2" /manifest ......targetnativeReleasehadoop.dll.intermediate.manifest
Deleting file "......targetnativeReleasenative.unsuccessfulbuild".
Touching "......targetnativeReleasenative.lastbuildstate".
Done Building Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.vcxproj" (default targets).
Done Building Project "C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.sln" (default targets).
Build succeeded.
"C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.sln" (default target) (1) ->
"C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.vcxproj" (default target) (2) ->
(ClCompile target) ->
srcorgapachehadoopsecurityJniBasedUnixGroupsMappingWin.c(48): warning C4047: 'function' : 'LPSTR' differs in levels of indirection from 'LPSTR
*' [C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.vcxproj]
srcorgapachehadoopsecurityJniBasedUnixGroupsMappingWin.c(48): warning C4024: 'FormatMessageA' : different types for formal and actual
parameter 5 [C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrcmainnativenative.vcxproj]
2 Warning(s)
0 Error(s)
Time Elapsed 00:00:04.47
[INFO]
[INFO] --- avro-maven-plugin:1.7.4:schema (generate-avro-test-sources) @ hadoop-common ---

[INFO]
[INFO] --- hadoop-maven-plugins:2.2.0:protoc (compile-test-protoc) @ hadoop-common ---
[INFO]
[INFO] --- maven-resources-plugin:2.2:testResources (default-testResources) @ hadoop-common ---
[INFO] Using default encoding to copy filtered resources.
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run (create-log-dir) @ hadoop-common ---
[INFO] Executing tasks
main:
[delete] Deleting directory C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontargettestdata
[mkdir] Created dir: C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontargettestdata
[mkdir] Created dir: C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontargetlog
[copy] Copying 6 files to C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontargettest-classes
[INFO] Executed tasks
[INFO]
[INFO] --- maven-antrun-plugin:1.6:run (copy-test-tarballs) @ hadoop-common ---
[INFO] Executing tasks
main:
[copy] Copying 2 files to C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontargettest-classes
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-common ---
[INFO] Compiling 341 source files to C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commontargettest-classes
[WARNING] C:Hadoophadoop-2.2.0-srchadoop-common-projecthadoop-commonsrctestjavaorgapachehadoopsecuritysslKeyStoreTestUtil.java:
[108,15] X509CertImpl is internal proprietary API and may be removed in a future release
[INFO]
[INFO] --- maven-surefire-plugin:2.12.3:test (default-test) @ hadoop-common ---
[INFO] Tests are skipped.
[INFO] --- maven-jar-plugin:2.3.1:jar (prepare-jar) @ hadoop-common ---
Apache Hadoop Main ................................ SUCCESS [2.839s]
[INFO] Apache Hadoop Project POM ......................... SUCCESS [1.389s]
[INFO] Apache Hadoop Annotations ......................... SUCCESS [2.867s]
[INFO] Apache Hadoop Assemblies .......................... SUCCESS [0.191s]
[INFO] Apache Hadoop Project Dist POM .................... SUCCESS [1.808s]
[INFO] Apache Hadoop Maven Plugins ....................... SUCCESS [3.997s]
[INFO] Apache Hadoop Auth ................................ SUCCESS [3.365s]
[INFO] Apache Hadoop Auth Examples ....................... SUCCESS [2.511s]
[INFO] Apache Hadoop Common .............................. SUCCESS [2:19.517s]
[INFO] Apache Hadoop NFS ................................. SUCCESS [7.271s]
[INFO] Apache Hadoop Common Project ...................... SUCCESS [0.052s]
[INFO] Apache Hadoop HDFS ................................ SUCCESS [1:48.356s]
[INFO] Apache Hadoop HttpFS .............................. SUCCESS [29.764s]

[INFO] Apache Hadoop HDFS BookKeeper Journal ............. SUCCESS [15.483s]
[INFO] Apache Hadoop HDFS-NFS ............................ SUCCESS [5.892s]
[INFO] Apache Hadoop HDFS Project ........................ SUCCESS [0.116s]
[INFO] hadoop-yarn ....................................... SUCCESS [0.755s]
[INFO] hadoop-yarn-api ................................... SUCCESS [58.341s]
[INFO] hadoop-yarn-common ................................ SUCCESS [41.616s]
[INFO] hadoop-yarn-server ................................ SUCCESS [0.222s]
[INFO] hadoop-yarn-server-common ......................... SUCCESS [12.719s]
[INFO] hadoop-yarn-server-nodemanager .................... SUCCESS [18.352s]
[INFO] hadoop-yarn-server-web-proxy ...................... SUCCESS [4.325s]
[INFO] hadoop-yarn-server-resourcemanager ................ SUCCESS [24.415s]
[INFO] hadoop-yarn-server-tests .......................... SUCCESS [0.457s]
[INFO] hadoop-yarn-client ................................ SUCCESS [6.260s]
[INFO] hadoop-yarn-applications .......................... SUCCESS [0.163s]
[INFO] hadoop-yarn-applications-distributedshell ......... SUCCESS [2.488s]
[INFO] hadoop-mapreduce-client ........................... SUCCESS [0.098s]
[INFO] hadoop-mapreduce-client-core ...................... SUCCESS [33.843s]
[INFO] hadoop-yarn-applications-unmanaged-am-launcher .... SUCCESS [2.359s]
[INFO] hadoop-yarn-site .................................. SUCCESS [0.175s]
[INFO] hadoop-yarn-project ............................... SUCCESS [1.226s]
[INFO] hadoop-mapreduce-client-common .................... SUCCESS [25.130s]
[INFO] hadoop-mapreduce-client-shuffle ................... SUCCESS [3.742s]
[INFO] hadoop-mapreduce-client-app ....................... SUCCESS [15.882s]
[INFO] hadoop-mapreduce-client-hs ........................ SUCCESS [8.723s]
[INFO] hadoop-mapreduce-client-jobclient ................. SUCCESS [7.939s]
[INFO] hadoop-mapreduce-client-hs-plugins ................ SUCCESS [2.220s]
[INFO] Apache Hadoop MapReduce Examples .................. SUCCESS [7.636s]
[INFO] hadoop-mapreduce .................................. SUCCESS [0.597s]
[INFO] Apache Hadoop MapReduce Streaming ................. SUCCESS [5.471s]
[INFO] Apache Hadoop Distributed Copy .................... SUCCESS [12.711s]
[INFO] Apache Hadoop Archives ............................ SUCCESS [2.580s]
[INFO] Apache Hadoop Rumen ............................... SUCCESS [9.520s]
[INFO] Apache Hadoop Gridmix ............................. SUCCESS [5.555s]
[INFO] Apache Hadoop Data Join ........................... SUCCESS [3.696s]
[INFO] Apache Hadoop Extras .............................. SUCCESS [4.345s]
[INFO] Apache Hadoop Pipes ............................... SUCCESS [0.067s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [1.350s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [0.055s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [15.598s]
[INFO] Apache Hadoop Client .............................. SUCCESS [5.230s]
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [0.438s]
[INFO] Total time: 11:14.078s
[INFO] Finished at: Sat Feb 13 19:05:29 IST 2016
[INFO] Final Memory: 80M/395M

C:cygwin64homehadoop-2.2.0etchadoophadoop-env.cmd
set JAVA_HOME=C:Javajdk1.7.0_80
Note:
make sure java home without space I.e avoid like c:/program files
C:cygwin64homehadoop-2.2.0etchadoopcore-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
C:cygwin64homehadoop-2.2.0etchadoopmapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
C:cygwin64homehadoop-2.2.0etchadoophdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/yarn_data/dfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/yarn_data/dfs/datanode</value>
</property>
</configuration>
NOTE:
dont give c:/cygwin it throws error whatever you give it takes the root from cygwin install directory.

C:cygwin64homehadoop-2.2.0etchadoopyarn-site.xml
<configuration>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>
%HADOOP_HOME%etchadoop,
%HADOOP_HOME%sharehadoopcommon*,
%HADOOP_HOME%sharehadoopcommonlib*,
%HADOOP_HOME%sharehadoopmapreduce*,
%HADOOP_HOME%sharehadoopmapreducelib*,
%HADOOP_HOME%sharehadoophdfs*,
%HADOOP_HOME%sharehadoophdfslib*,
%HADOOP_HOME%sharehadoopyarn*,
%HADOOP_HOME%sharehadoopyarnlib*
</value>
</property>
</configuration>
16/02/14 13:51:29 INFO service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl failed in state INITED;
cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize LocalizationServiceorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize Loc
alizationService
cygwin output:
ERROR:
if you run hadopp on cygwin it won't work. Please work on windows cmd it works
************************************************************/
16/02/13 19:40:14 WARN common.Util: Path /yarn_data/dfs/datanode should be speci
fied as a URI in configuration files. Please update hdfs configuration.
16/02/13 19:40:15 WARN util.NativeCodeLoader: Unable to load native-hadoop libra
ry for your platform... using builtin-java classes where applicable
16/02/13 19:40:17 FATAL datanode.DataNode: Exception in secureMain
java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.a
ccess0(Ljava/lang/String;I)Z
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
at org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:977)
at org.apache.hadoop.util.DiskChecker.checkAccessByFileMethods(DiskCheck

Jars need to run
avro-1.7.4.jar
commons-configuration-1.6.jar
commons-httpclient-3.1.jar
commons-lang-2.5.jar
commons-logging-1.1.1.jar
guava-11.0.2.jar
hadoop-auth-2.2.0.jar
hadoop-common-2.2.0.jar
hadoop-hdfs-2.2.0.jar
hadoop-mapreduce-client-common-2.2.0.jar
hadoop-mapreduce-client-core-2.2.0.jar
hadoop-nfs-2.2.0.jar
hadoop-yarn-api-2.2.0.jar
hadoop-yarn-common-2.2.0.jar
jackson-core-asl-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
log4j-1.2.17.jar
slf4j-api-1.7.5.jar

c:usersadmin>hdfs dfs -mkdir /input

see the output
C:Usersadmin>yarn jar C:/HADOOPOUTPUT/mapreduce.jar test.WordCount /input/wordcount.txt /output
hadoopmapreduce in ubuntu commands
client-core-2.5.jar in class path.
hdadmin@ubuntu:/jars$ hadoop fs -ls
hdadmin@ubuntu:/jars$ hadoop fs -mkdir /arun
create a directory arun in hdfs access by hdfs://arun
hdadmin@ubuntu:/jars$ hadoop fs -copyFromLocal sales.csv /arun/salesArun.csv
copy a file from /home/hdadmin/jars/sales.csv to hdfs with new name /arun/salesArun.csv
hdadmin@ubuntu:/jars$ hadoop jar firsthadoop.jar mapreduce.SalesCountryDriver
/arun/salesArun.csv /output
run the hadoop mapreduce jar with name firsthadoop with main class SalesCountry Driver
program
All the default port no will be inside /HADOOP_HOME/share/doc/hadoop webapps

To force safe mode:
hadoop namenode -recover -force
temp files files will be under c:/tmp/hadoop-admin directory
To change tmp Directory
To change the local temp directory use:
-D dfs.data.dir=/tmp
To specify additional local temp directories use:
-D mapred.local.dir=/tmp/local
-D mapred.system.dir=/tmp/system
-D mapred.temp.dir=/tmp/temp
Hadoop Streaming

Windowshadoop

More Related Content

What's hot

Viewers also liked

Similar to Windowshadoop

Recently uploaded

Windowshadoop