MarioNote

June 4, 2010

Hadoop Installation Note (Standalone)

Filed under: Java,Linux,step-by-step — mariojump @ 10:30 pm
Tags: ,

This is a note when I installed Hadoop on a single machine (standalone).

I followed Hadoop Quick Start available at:
http://hadoop.apache.org/common/docs/r0.20.0/quickstart.html

Check Java version — Java 1.6.x is required, and Sun version is preferred. See:
http://hadoop.apache.org/common/docs/r0.20.0/quickstart.html

Download hadoop-0.20.2.tar.gz from:
http://hadoop.apache.org/mapreduce/releases.html#Download

$ tar xvzf hadoop-0.20.2.tar.gz

Skip rsync and ssh installation, as they are already available on the machine.

// Check the command usage:
$ cd hadoop-0.20.2
$ bin/hadoop
Usage: hadoop [–config confdir] COMMAND
where COMMAND is one of:
namenode -format format the DFS filesystem
secondarynamenode run the DFS secondary namenode
namenode run the DFS namenode
datanode run a DFS datanode
dfsadmin run a DFS admin client
mradmin run a Map-Reduce admin client
fsck run a DFS filesystem checking utility
fs run a generic filesystem user client
balancer run a cluster balancing utility
jobtracker run the MapReduce job Tracker node
pipes run a Pipes job
tasktracker run a MapReduce task Tracker node
job manipulate MapReduce jobs
queue get information regarding JobQueues
version print the version
jar run a jar file
distcp copy file or directories recursively
archive -archiveName NAME * create a hadoop archive
daemonlog get/set the log level for each daemon
or
CLASSNAME run the class named CLASSNAME
Most commands print help when invoked w/o parameters.

$ mkdir input
$ cp conf/*.xml input
$ bin/hadoop jar hadoop-*-examples.jar grep input output ‘dfs[a-z.]+’
Error: JAVA_HOME is not set.

$ /usr/java/latest/bin/java -version
java version “1.6.0_16”

$ export JAVA_HOME=/usr/java/latest
[added this line in the .bashrc]
Reference: http://www.atmarkit.co.jp/fjava/special/distributed03/distributed03_1.html

Try again:
$ bin/hadoop jar hadoop-*-examples.jar grep input output ‘dfs[a-z.]+’

No error message is shown this time.
$ cat output/*
1 dfsadmin

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: