Hadoop Commands – HDFS dfs commands, Hadoop Linux commands

Hadoop commands list is a lot bigger than the list demonstrated here, however, we have explained some of the very useful Hadoop commands below.

“hadoop fs” lists all the Hadoop commands that can be run in FsShell

“hadoop fs -help ” will display help for that command where is the actual name of the command.

Hadoop Commands and HDFS Commands

All HDFS commands are invoked by the “bin/hdfs” script. If we will run the hdfs scripts without any argument then it will print the description of all commands.

CommandsUsagesDescription
classpathhdfsclasspathIt prints the class path needed to get the Hadoop jar and the required libraries.
lshadoop fs -ls /List the contents of the root directory in HDFS
versionhadoop versionPrint the Hadoop version
dfhadoop fs -dfhdfs:/amount of space used and available on currently mounted filesystem
balancerhadoop balancerRun a cluster balancing utility
mkdirhadoop fs -mkdir /usr/training/hadoop_filesCreate a new directory hadoop_files below the /usr/training directory in HDFS
puthadoop fs -put /data/myfile.txt /usr/training/hadoop_filesAdd a sample text file from the unix local directory (/data/myfile.txt) to the HDFS directory /usr/training/hadoop_files
lshadoop fs -ls /usr/training/hadoop_filesList the contents of this new directory in HDFS.
puthadoop fs -put /data/finance /usr/training/hadoop_filesAdd the entire local unix directory to HDFS filesystem (/usr/training/hadoop_files)
duhadoop fs -du -s -h hadoop_files/financeSee how much space a given directory occupies in HDFS.
rmhadoop fs -rm hadoop/finance/myfile.txtDelete a file “myfile.txt” from the “finance” directory.
rmhadoop fs -rm hadoop_files/finance/*Delete all files from the “finance” directory using a wildcard.
expungehadoop fs –expungeTo empty the trash
cathadoop fs -cat hadoop_files/myfile.txtSee the content of “myfile.txt” present in /hadoop_file directory
copyToLocalhadoop fs -copyToLocalhadoop_files/myfile.txt /scratch/dataAdd the myfile.txt file from “hadoop_files” directory which is present in HDFS directory to the directory “data” which is present in your local directory
gethadoop fs -get hadoop_files/myfile.txt /scratch/dataget command can be used alternaively to “copyToLocal” command
chmodsudo -u hdfs hadoop fs -chmod 600 hadoop_files/myfiles.txt Use “-chmod” command to change permissions of a file. Default file permissions are 666 in HDFS
mvhadoop fs -mv hadoop_filesapache_hadoopMove a directory from one location to othe
expungehadoop fs -expungeCommand to make the name node leave safe mode

Hadoop fs commands – HDFS dfs commands

CommandsUsagesDescription
fshadoop fsList all the Hadoop file system shell commands
helphadoop fs –helpHelp for any command
TOUCHZhdfs dfs –touchz /hadoop_files/myfile.txtCreate a file in HDFS with file size 0 bytes
rmrhdfs dfs –rmr /hadoop_files/Remove the directory to HDFS
counthdfs dfs –count /userCount the number of directories, files, and bytes under the paths that match the specified file pattern.

Hadoop Linux commands

CommandExampleDescription
lsls -l

ls -a

ls -l /etc

Lists files in current directory.If you run ls without any additional parameters, the program will list the contents of the current directory in short form.
-l
detailed list
-a
displays hidden files
cpcp [option(s)] <sourcefile> <targetfile>

cp file1 new-file2

cp -r dir1 dir2

Copies sourcefile to targetfile.
-i
Waits for confirmation, if necessary, before an existing targetfile is overwritten
-r
Copies recursively (includes subdirectories)
mv$ mv file_1.txt /scratch/kmakMove or rename files. Copies sourcefile to targetfile then deletes the original sourcefile.
rmrm myfile.txt

rm -r mydirectory

Removes the specified files from the file system. Directories are not removed by rm unless the option -r is used.
lnln file1.txt file2.txtln creates links between files.
cdcd  /scratch/kmak/biChanges the shell’s current working directory.
pwdpwdPrint working directory.It writes the full pathname of the current working directory to the standard output.
mkdirmkdir <mydir>It is used to create directories on a file system.
rmdirrmdir <emptydir>Deletes the specified directory provided it is already empty.
nlnl myfile.txtnl numbers the lines in a file.
geditgedit myfile.txtText editor
statstat myfile.txtDisplays the status of an entire file system.
wcwc myfile.txt
wc -l myfile.txt
wc -c myfile.txt
It is used to find out the number of newline count, word count, byte, and characters count in a file specified by the file arguments.
chownchown chope file.txt
chown -R chope /scratch/work
It changes the owner and owning group of files.
chgrpchgrp oracle myfile.txtChanges group ownership of a file or files.
ifconfigIfconfigIt is used to view and change the configuration of the network interfaces on your system.

Team RCV Academy

About Team RCV Academy

RCV Academy Team is a group of professionals working in various industries and contributing to tutorials on the website and other channels.

View all posts by Team RCV Academy