Linux command line 4 - Redirection
This article mainly references the book The Linux Command Line (2nd Edition). The shell tool used is Xshell, and the operating system is CentOS 7.6. This means that some shortcuts may not work in other shell tools, and there may be slight differences in files across different Linux operating systems. Please analyze based on your actual situation.
Since CentOS 7.6 is EOL, consider using Rocky Linux or Alma Linux instead.
I/O Redirection
“I/O” stands for Input/Output. By using I/O redirection, you can specify a file as the source of input or the destination for output from a command. Additionally, you can connect multiple commands together to form a more powerful command.
Standard Input, Output, and Error
So far, many of the programs we’ve used generate some type of output. This output usually comes in two types. The first type is the result of the program’s execution, i.e., what it accomplishes. The second type is status and error messages, which inform us about the program’s progress. If we observe commands like ls, we will see both its execution results and error messages displayed on the screen.
In keeping with the Unix theme that “everything is a file,” programs like ls actually send their execution results to a special file called standard output (usually denoted as stdout), while their status information is sent to another file called standard error (stderr). By default, both standard output and standard error are connected to the screen, not saved to a disk file.
Additionally, many programs receive input from a device called standard input (stdin), which, by default, is connected to the keyboard.
Standard Output Redirection
>: Redirection symbol
We can use the > redirection symbol followed by a file name to redirect standard output to a file other than the screen.
For example, we can instruct the shell to send the result of the ls command to the file ls-output.txt:
1 | [root@lht ~]# ll /usr/bin > ls-output.txt |
At this point, all the output information is written to the file, so nothing is displayed on the screen.
Let’s take a look at the output file:
1 | [root@lht ~]# ll ls-output.txt |
1 | [root@lht ~]# less ls-output.txt |
Now, let’s switch to a non-existent directory
1 | [root@lht ~]# ll /bin/usr > ls-output.txt |
As expected, we received an error message.
The error message appears on the screen instead of being written to the file because we only redirected standard output, and ls outputs the error message to standard error, so it is not written to the file but displayed on the screen.
Let’s check the output file again.
1 | [root@lht ~]# ll ls-output.txt |
At this point, the file length has become 0. This is because when using the > redirection symbol to redirect output, the target file is always overwritten from the beginning. Since there was an error here, the output is 0, and the file content was cleared.
If we need to clear the contents of a file (or create a new empty file), we can use the following command:
1 | [root@lht ~]# > emptyfile |
If we want to append the result to the end of the file instead of overwriting the original content, we need to use another redirection symbol— >>
1 | [root@lht ~]# ll >> emptyfile |
After executing it three times, the file size is three times that of the first execution.
Standard Error Redirection
A program can produce output on any of several numbered file streams. While we refer to the first three of these streams as standard input, output, and error, the shell internally refers to them as file descriptors 0, 1, and 2. The shell uses file descriptors to provide a way to redirect files. Since standard error is the same as file descriptor 2, we use this notation to redirect standard error:
1 | [root@lht ~]# ll /bin/usr 2> ls-error.txt |
The file descriptor “2”, placed right before the redirection operator, accomplishes the task of redirecting standard error to the file ls-error.txt.
Let’s check the output file:
1 | [root@lht ~]# cat ls-error.txt |
Redirecting Standard Output and Error to the Same File
There are two methods:
- Traditional Method (works in older shells as well)
1 | [root@lht ~]# ll /bin/usr > ls-output.txt 2>&1 |
This actually performs two redirections: first, standard output is redirected to the file ls-output.txt, and then file descriptor 2 (standard error) is redirected to file descriptor 1 (standard output) using the notation 2>&1.
Note that the order of redirections is crucial. The redirection of standard error must always appear after the redirection of standard output, otherwise it won’t work. Here is an incorrect example:
1 | [root@lht ~]# ll /bin/usr 2>&1 > ls-output.txt |
In this case, standard error is still displayed on the screen.
- Newer Bash Versions provide a second, more concise method:
1 | [root@lht ~]# ll /bin/usr &> ls-output.txt |
Using just one notation, &>, redirects both standard output and standard error to the file ls-output.txt.
Handling Unwanted Output
Sometimes we don’t want the output of a command and just want to discard it. This is particularly useful for error and status messages. We can redirect the output to a special file called “/dev/null”, which is a system device known as a bit bucket. It accepts input and discards it.
1 | [root@lht ~]# ll /bin/usr 2> /dev/null |
Unix Culture and
/dev/nullThe bit bucket is an ancient concept in Unix. Because of its universality, it appears in many parts of Unix culture.
When someone says they are sending your comments to
/dev/null, now you know what that means.For more examples, you can read the Wikipedia article about “/dev/null”.
cat – Concatenate Files
We introduce the cat command here to prepare for input redirection.
The cat command reads one or more files and then copies them to standard output.
1 | [root@lht ~]# cat ls-output.txt |
cat is often used to display short text files. Since cat can accept more than one file as an argument, it can also be used to concatenate files together.
For example, if we download a large file that has been split into several parts (multimedia files on USENET are often separated this way), we can use cat to join them together.
1 | cat movie.mpeg.0* > movie.mpeg |
Since wildcards always expand in order, these parts will be merged in the correct sequence.
Standard Input Redirection
If we use the cat command without any arguments, it waits to read input from standard input. Since standard input is connected to the keyboard by default, it’s waiting for us to input data!
We can type in text to input content and then press Ctrl-D to indicate the end of input. At this point, the cat command will copy our input to standard output, so we’ll see the text repeated.
1 | [root@lht ~]# cat |
After pressing Ctrl-D:
1 | [root@lht ~]# cat |
Alternatively, if we press Enter after inputting the text, the repeated content will appear on the next line:
1 | [root@lht ~]# cat |
Now, we can redirect standard input:
1 | [root@lht ~]# cat < ls-output.txt |
Here, we redirected standard input from the keyboard to a file. This example simply demonstrates how to use a file as the source of standard input. Other commands make better use of standard input, which we will cover later.
Pipelines
By using the pipeline operator | (vertical bar), the standard output of one command can be passed through a pipeline to the standard input of another command.
The format is as follows:
1 | command1 | command2 |
Example:
We can use less to display the output of a command page by page:
1 | [root@lht ~]# ll /usr/bin | less |
Filters
Due to the nature of pipelines, we can combine multiple commands together. This is referred to as filters.
Using the previous command as an example, now we want to sort the output. We can use the sort command to sort the output:
1 | [root@lht ~]# ls /bin /usr/local | sort | less |
Here, we specified two directories, then used the sort command to sort the output of ls, and finally passed the data to less for viewing.
uniq – Report or Omit Repeated Lines
The uniq command is often used in combination with sort. By default, uniq removes any repeated lines.
1 | [root@lht ~]# ls /bin/ /usr/bin/ | sort | uniq | less |
1 | a2p |
In the results, there are no duplicates. In fact, /bin and /usr/bin are the same directory. /bin is a symbolic link that points to /usr/bin, so the content is the same, but the uniq command removed the duplicates.
1 | [root@lht ~]# ll /bin |
If we want to display only duplicate entries, we can use the d option with the uniq command, which will show only the repeated data.
1 | [root@lht ~]# ls /bin/ /usr/bin/ | sort | uniq -d | less |
1 | [ |
Use the --help option for more parameters and usage, just like we mentioned earlier.
wc – Print Line, Word, and Byte Counts
The wc (word count) command can be used to count the number of newlines, words, and bytes in a file. For example:
1 | [root@lht ~]# wc ls-output.txt |
The output shows three numbers, which represent the line count, word count, and byte count of ls-output.txt.
As with previous commands, if no command-line arguments are specified, the wc command will read from standard input.
The -l option restricts the output to only the line count of the file. When piped, it allows us to know how many lines are in an ordered list, like this:
1 | [root@lht ~]# ls /bin/ /usr/bin/ | sort | uniq | wc -l |
grep – Output Lines Matching a Pattern
The grep command is used to search for text patterns in files, with the syntax:
1 | grep pattern filename |
In simple terms, the grep command outputs only the results that match the specified pattern.
The grep command can match very complex patterns, and you may have thought about using regular expressions; we will cover that later. For now, let’s look at some simple examples.
Suppose we want to know how many files in the /bin directory contain the word “zip” to determine how many programs and files are related to compression. We can use the following command:
1 | [root@lht ~]# ls /bin/ | sort | uniq | grep zip |
The grep command has some very convenient options:
-i: Makes thegrepcommand ignore case distinctions when searching (it is case-sensitive by default).-v: Makes thegrepcommand output only the lines that do not match the specified pattern.-c: Makes thegrepcommand output only the count of lines that match the specified pattern.
head/tail – Output the Beginning/End of Files
Sometimes, we don’t want the full output of a command or file, but just the first few lines or last few lines.
The head and tail commands output the first and last 10 lines of a file by default, respectively, but the number of lines can be adjusted using the -n option.
1 | [root@lht ~]# head -n 5 ls-output.txt |
Both commands can be used in a pipeline:
1 | [root@lht ~]# ls /bin/ | tail -n 5 |
The tail command has an option to view files in real-time. This option is suitable for monitoring logs as they are written.
In the following example, we will view the messages file in the /var/log directory (if the messages file does not exist, use /var/log/syslog instead). Since the /var/log/messages file may contain security information, superuser privileges may be required to execute this operation on some Linux distributions:
1 | [root@lht ~]# tail -f /var/log/messages |
By using the -f option, the tail command will continuously monitor the file, and any new lines added will immediately appear on the screen. This process will continue until you stop it by pressing Ctrl-C.
tee – Read Standard Input and Write Output to Standard Output and Files
To maintain the analogy of pipelines, Linux also provides a command called tee, which acts as a “T-junction” in a pipeline.
The tee command reads content from standard input and copies it to standard output (allowing the data to continue flowing down the pipeline) and to other files. This is useful for capturing data at a certain intermediate stage in the processing of a pipeline.
We will reuse a previous example here, this time adding the tee command before the grep command to save the entire directory list to ls.txt:
1 | [root@lht ~]# ls /bin/ | tee ls.txt | grep zip |
- Title: Linux command line 4 - Redirection
- Author: LHT
- Created at : 2023-01-04 08:00:00
- Link: https://blog.327774.xyz/2023/01/04/linux/Linux command line 4 - Redirection/
- License: This work is licensed under CC BY-NC-SA 4.0.