Hadoop commands list is a lot bigger than the list demonstrated here, however, we have explained some of the very useful Hadoop commands below.
“hadoop fs” lists all the Hadoop commands that can be run in FsShell
“hadoop fs -help ” will display help for that command where is the actual name of the command.
Hadoop Commands and HDFS Commands
All HDFS commands are invoked by the “bin/hdfs” script. If we will run the hdfs scripts without any argument then it will print the description of all commands.
| Commands | Usages | Description |
| classpath | hdfsclasspath | It prints the class path needed to get the Hadoop jar and the required libraries. |
| ls | hadoop fs -ls / | List the contents of the root directory in HDFS |
| version | hadoop version | Print the Hadoop version |
| df | hadoop fs -dfhdfs:/ | amount of space used and available on currently mounted filesystem |
| balancer | hadoop balancer | Run a cluster balancing utility |
| mkdir | hadoop fs -mkdir /usr/training/hadoop_files | Create a new directory hadoop_files below the /usr/training directory in HDFS |
| put | hadoop fs -put /data/myfile.txt /usr/training/hadoop_files | Add a sample text file from the unix local directory (/data/myfile.txt) to the HDFS directory /usr/training/hadoop_files |
| ls | hadoop fs -ls /usr/training/hadoop_files | List the contents of this new directory in HDFS. |
| put | hadoop fs -put /data/finance /usr/training/hadoop_files | Add the entire local unix directory to HDFS filesystem (/usr/training/hadoop_files) |
| du | hadoop fs -du -s -h hadoop_files/finance | See how much space a given directory occupies in HDFS. |
| rm | hadoop fs -rm hadoop/finance/myfile.txt | Delete a file “myfile.txt” from the “finance” directory. |
| rm | hadoop fs -rm hadoop_files/finance/* | Delete all files from the “finance” directory using a wildcard. |
| expunge | hadoop fs –expunge | To empty the trash |
| cat | hadoop fs -cat hadoop_files/myfile.txt | See the content of “myfile.txt” present in /hadoop_file directory |
| copyToLocal | hadoop fs -copyToLocalhadoop_files/myfile.txt /scratch/data | Add the myfile.txt file from “hadoop_files” directory which is present in HDFS directory to the directory “data” which is present in your local directory |
| get | hadoop fs -get hadoop_files/myfile.txt /scratch/data | get command can be used alternaively to “copyToLocal” command |
| chmod | sudo -u hdfs hadoop fs -chmod 600 hadoop_files/myfiles.txt | Use “-chmod” command to change permissions of a file. Default file permissions are 666 in HDFS |
| mv | hadoop fs -mv hadoop_filesapache_hadoop | Move a directory from one location to othe |
| expunge | hadoop fs -expunge | Command to make the name node leave safe mode |
Hadoop fs commands – HDFS dfs commands
| Commands | Usages | Description |
| fs | hadoop fs | List all the Hadoop file system shell commands |
| help | hadoop fs –help | Help for any command |
| TOUCHZ | hdfs dfs –touchz /hadoop_files/myfile.txt | Create a file in HDFS with file size 0 bytes |
| rmr | hdfs dfs –rmr /hadoop_files/ | Remove the directory to HDFS |
| count | hdfs dfs –count /user | Count the number of directories, files, and bytes under the paths that match the specified file pattern. |
Hadoop Linux commands
| Command | Example | Description |
| ls | ls -l ls -a ls -l /etc | Lists files in current directory.If you run ls without any additional parameters, the program will list the contents of the current directory in short form. -l detailed list -a displays hidden files |
| cp | cp [option(s)] <sourcefile> <targetfile> cp file1 new-file2 cp -r dir1 dir2 | Copies sourcefile to targetfile. -i Waits for confirmation, if necessary, before an existing targetfile is overwritten -r Copies recursively (includes subdirectories) |
| mv | $ mv file_1.txt /scratch/kmak | Move or rename files. Copies sourcefile to targetfile then deletes the original sourcefile. |
| rm | rm myfile.txt rm -r mydirectory | Removes the specified files from the file system. Directories are not removed by rm unless the option -r is used. |
| ln | ln file1.txt file2.txt | ln creates links between files. |
| cd | cd /scratch/kmak/bi | Changes the shell’s current working directory. |
| pwd | pwd | Print working directory.It writes the full pathname of the current working directory to the standard output. |
| mkdir | mkdir <mydir> | It is used to create directories on a file system. |
| rmdir | rmdir <emptydir> | Deletes the specified directory provided it is already empty. |
| nl | nl myfile.txt | nl numbers the lines in a file. |
| gedit | gedit myfile.txt | Text editor |
| stat | stat myfile.txt | Displays the status of an entire file system. |
| wc | wc myfile.txt wc -l myfile.txt wc -c myfile.txt | It is used to find out the number of newline count, word count, byte, and characters count in a file specified by the file arguments. |
| chown | chown chope file.txt chown -R chope /scratch/work | It changes the owner and owning group of files. |
| chgrp | chgrp oracle myfile.txt | Changes group ownership of a file or files. |
| ifconfig | Ifconfig | It is used to view and change the configuration of the network interfaces on your system. |