Linux command line 5 - Seeing the World as the Shell Sees It

This article mainly references the book The Linux Command Line (2nd Edition). The shell tool used is Xshell, and the operating system is CentOS 7.6. This means that some shortcuts may not work in other shell tools, and there may be slight differences in files across different Linux operating systems. Please analyze based on your actual situation.

Since CentOS 7.6 is EOL, consider using Rocky Linux or Alma Linux instead.

Viewing the World from the Shell’s Perspective

Character Expansion

Each time you input a command and press the Enter key, Bash performs some text substitution operations before executing the command.

We’ve already encountered a few examples where simple characters like * have various meanings in the Shell. This process of handling is called expansion.

The content we input is expanded into other content before being executed by the Shell. To illustrate this, let’s take a look at the echo command.

echo: Displays a line of text.

[root@lht ~]# echo this is a test
this is a test

The arguments passed to the echo command are displayed one by one. Let’s look at another example:

[root@lht ~]# echo *
example ls-output.txt ls.txt
[root@lht ~]# ls
example  ls-output.txt  ls.txt

Why was * not outputted? Recall the usage of wildcards: * means to match any number of characters in a filename, but we haven’t discussed how the Shell implements that functionality.

The simple answer is: before executing the echo command, the Shell expands * into other content (in this case, the filenames in the current working directory). When you press Enter, the Shell automatically expands all qualifying characters in the command line before executing the command.

Therefore, the echo command never receives *; it only receives the expanded result. Knowing this, we can see that the behavior of the echo command is entirely as expected.

Pathname Expansion

The working mechanism of wildcards is called pathname expansion.

Here are some examples:

[root@lht ~]# ls
documents  example  ls-output.txt  ls.txt  pictures  programs  videos
[root@lht ~]# echo ls*
ls-output.txt ls.txt
[root@lht ~]# echo *s
documents pictures programs videos

You can even execute the following command:

[root@lht ~]# echo *[[:lower:]]
documents example ls-output.txt ls.txt pictures programs videos

This displays files that end with a lowercase letter.

Or check the contents of another directory:

[root@lht usr]# echo /usr/*bin
/usr/bin /usr/sbin

Have you noticed that using the wildcard * does not list hidden files? If you want to list all hidden files but don’t want the result to include . and .., which represent the current and parent directories, you can write:

echo .[!.]*

This will display files where the first character is ., but the second character is not ., which can handle most hidden files (though it won’t show files starting with multiple dots).

The A option of the ls command can correctly list hidden files:

ls -A

Tilde Expansion

As we mentioned when introducing the cd command, the tilde ~ has a special meaning.

When it precedes a word, it expands to the home directory of the named user. If no word is specified, it expands to the current user’s home directory:

[root@lht home]# echo ~
/root
[root@lht home]# echo ~halo
/home/halo

Arithmetic Expansion

Shell performs arithmetic operations through expansion, as illustrated below:

[root@lht example]# echo $((2 + 2))
4

Format:

$((expression))

Here, expression is an arithmetic expression composed of numbers and arithmetic operators.

Arithmetic expressions only support integers (not decimals), but they can perform various operations.

Here are some of the supported arithmetic operators:

Operator	Description
+	Addition
-	Subtraction
*	Multiplication
/	Division (results in an integer)
%	Modulus (remainder)
**	Exponentiation

In arithmetic expressions, spaces are not significant. For example, five squared multiplied by three:

[root@lht example]# echo $((5**2*3))
75

Here’s an example using division and the modulus operator. Note the result of integer division:

[root@lht example]# echo Five divided by two equals $((5/2))
Five divided by two equals 2
[root@lht example]# echo with $((5%2)) left over.
with 1 left over.

We will discuss arithmetic expressions in more detail later.

Brace Expansion

You can create multiple text strings according to the pattern within braces:

[root@lht example]# echo Front-{A,B,C}-Back Front-A-Back
Front-A-Back Front-B-Back Front-C-Back Front-A-Back

The patterns used for brace expansion can include a prefix (preamble) and a suffix (postscript).

Brace expressions themselves can be a list of comma-separated strings or a range of integers or single characters, but they cannot contain unquoted whitespace characters. Here’s an example using an integer range:

[root@lht example]# echo Number_{1..5}
Number_1 Number_2 Number_3 Number_4 Number_5

In Bash 4.0 and above, integers can also be zero-padded:

[root@lht example]# echo {01..15}
01 02 03 04 05 06 07 08 09 10 11 12 13 14 15
[root@lht example]# echo {001..15}
001 002 003 004 005 006 007 008 009 010 011 012 013 014 015

Here’s a series of letters arranged in descending order:

[root@lht example]# echo {Z..A}
Z Y X W V U T S R Q P O N M L K J I H G F E D C B A

Brace expansion can also be nested:

[root@lht example]# echo a{A{1..4},B{5..8}}b
aA1b aA2b aA3b aA4b aB5b aB6b aB7b aB8b

So, what is the use of brace expansion? A common application is to create lists of files or directories.

Suppose you are a photographer with a large number of photos and want to organize them by year and month. The first thing to do is create a series of directories with a “year-month” naming format.

This way, the directories will be arranged in chronological order. You could manually type and create the entire directory list, but that would be tedious and error-prone. Instead, we can do it this way:

[root@lht ~]# mkdir photos
[root@lht ~]# cd photos/
[root@lht photos]# mkdir {2020..2023}-{01..12}
[root@lht photos]# ls
2020-01  2020-06  2020-11  2021-04  2021-09  2022-02  2022-07  2022-12  2023-05  2023-10
2020-02  2020-07  2020-12  2021-05  2021-10  2022-03  2022-08  2023-01  2023-06  2023-11
2020-03  2020-08  2021-01  2021-06  2021-11  2022-04  2022-09  2023-02  2023-07  2023-12
2020-04  2020-09  2021-02  2021-07  2021-12  2022-05  2022-10  2023-03  2023-08
2020-05  2020-10  2021-03  2021-08  2022-01  2022-06  2022-11  2023-04  2023-09

Parameter Expansion

Here, we will briefly introduce parameter expansion, which will be explained in more detail later when discussing shell scripts, as this feature is more practical in shell scripting than in command-line use.

Its many functions are related to the system’s ability to store a small amount of data and name various pieces of data. Several data elements (more accurately called variables) are available for users to view.

For example, your username can be stored in a variable named USER. To perform parameter expansion and display the content of USER, you can use the following command:

[root@lht ~]# echo $USER
root

If you want to see a list of available variables, you can do it this way:

[root@lht ~]# printenv | less

If you mistype the variable name, the output will be an empty string:

[root@lht ~]# echo $SUER

[root@lht ~]#

Command Substitution

You can use the output of a command as the content for expansion:

[root@lht ~]# echo $(ls)
example ls-output.txt ls.txt photos

In practical use, you might do the following:

[root@lht ~]# ll $(which cd)
-rwxr-xr-x. 1 root root 26 Oct 31  2018 /usr/bin/cd

This allows you to retrieve the detailed information of the cd command without knowing its full path.

The command mentioned in the original text:

ls -l $(which cp)

In practice, it results in an error:

[root@lht ~]# ll $(which cp)
ls: invalid option -- '''
Try 'ls --help' for more information.

The error message indicates an illegal option '. To check the cp command and see the cd command:

[root@lht ~]# which cp
alias cp='cp -i'
	/usr/bin/cp
[root@lht ~]# which cd
/usr/bin/cd

It shows that the cp command is actually cp -i. Since it contains a single quote, the ls command reports an illegal option, while the cd command is just a path, allowing it to run normally.

It is speculated that the original text’s which cp output was /bin/cp, which may be due to differences in system versions. This will not be explored further.

Command substitution is not limited to simple commands; entire pipelines can be used for substitution.

[root@lht ~]# file $(ls /bin/* | grep zip)
/bin/gpg-zip: POSIX shell script, ASCII text executable
/bin/gunzip:  POSIX shell script, ASCII text executable
/bin/gzip:    ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=526d77ff7164870f948d8f97aaf0a888cc561b30, stripped

In older versions of Shell, there is another syntax for command substitution that Bash also supports. In this syntax, backticks (```) are used instead of $(...):

[root@lht ~]# ll `which cd`
-rwxr-xr-x. 1 root root 26 Oct 31  2018 /usr/bin/cd

Quoting

As mentioned earlier, Shell has multiple expansion methods, and now we’ll discuss how to control these expansions.

Let’s look at a few examples:

[root@lht ~]# echo this is a   test
this is a test
[root@lht ~]# echo the total is $100.0
the total is 00.0

In the first example, Shell uses word splitting to remove the extra spaces in the parameter list of the echo command. In the second example, parameter expansion replaces $1 with an empty string because 1 is an undefined variable.

Shell provides a mechanism called quoting to selectively prevent unwanted expansions.

Double Quotes

The first type of quoting is double quotes. If text is placed within double quotes, all special characters used by Shell lose their special meaning and are treated as ordinary characters, except for $ (dollar sign), \ (backslash), and ` (backtick).

This means that word splitting, pathname expansion, tilde expansion, and brace expansion are all disabled, but parameter expansion, arithmetic expansion, and command substitution remain available.

With double quotes, we can handle filenames that contain whitespace. For instance, if we encounter a file named two words.txt, using that filename on the command line would cause word splitting to treat it as two separate arguments instead of the single argument we want:

[root@lht ~]# ll two words.txt
ls: cannot access two: No such file or directory
ls: cannot access words.txt: No such file or directory

By using double quotes, we prevent word splitting and obtain the desired result. Moreover, we can even fix the error:

[root@lht ~]# ll "two words.txt"
-rw-r--r-- 1 root root 0 Jun 18 17:09 two words.txt
[root@lht ~]# mv "two words.txt" two_words.txt

Remember, within double quotes, parameter expansion, arithmetic expression expansion, and command substitution remain effective:

[root@lht ~]# echo "$USER $((2+2)) $(cal)"
root 4       June 2023     
Su Mo Tu We Th Fr Sa
             1  2  3
 4  5  6  7  8  9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30

Earlier, we mentioned that word splitting removes extra spaces. Now let’s see how this works:

[root@lht ~]# echo this is a   test
this is a test

By default, word splitting treats spaces, tabs, and newlines as delimiters between words, meaning that unquoted spaces are not considered text but merely as separators.

In the above command, we have one command and four parameters; if we add double quotes, the spaces between the words are no longer treated as delimiters but as part of the parameters:

[root@lht ~]# echo "this is a   test"
this is a   test

Note that newline characters are treated as delimiters by the word-splitting mechanism, which has subtle implications for command substitution. Consider the following example:

[root@lht ~]# echo $(cal)
June 2023 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
[root@lht ~]# echo "$(cal)"
June 2023     
Su Mo Tu We Th Fr Sa
       1  2  3
4  5  6  7  8  9 10
11 12 13 14 15 16 17
18 19 20 21 22 23 24
25 26 27 28 29 30

In the example above, unquoted command substitution results in the output’s spaces and newlines being treated as delimiters, whereas using the quoting format produces correctly formatted output.

Single Quotes

To disable all expansions, single quotes should be used. The following compares the effects of no quotes, single quotes, and double quotes:

[root@lht ~]# echo text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER
text /root/ls-output.txt /root/two_words.txt a b foo 4 root
[root@lht ~]# echo "text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER"
text ~/*.txt {a,b} foo 4 root
[root@lht ~]# echo 'text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER'
text ~/*.txt {a,b} $(echo foo) $((2+2)) $USER

As the level of quoting increases, more expansions are disabled.

Escape Characters

Sometimes, we only want to quote a single character. To do this, we can precede the character with a backslash \, which is also known as an escape character. In double quotes, escape characters are often used to selectively avoid expansions:

[root@lht ~]# echo "The balance for user $USER is: \$5.00"
The balance for user root is: $5.00

Special characters that may appear in filenames (including $, !, &, and spaces) can have special meanings for Shell. To correctly include such special characters in a filename, escape characters are required:

[root@lht ~]# mv bad\&filename good_filename

If you want to input a backslash \, you need to use an escape character to escape itself, like this: \\.

[root@lht ~]# echo \\
\

In addition to being used as an escape character, a backslash can also be used to describe certain special characters known as control codes.

The first 32 characters in the ASCII encoding scheme are used to send commands to devices like teletypes. Some of these characters are quite familiar (such as tab, backspace, newline, and carriage return), while others are used less frequently.

Escape Sequence	Meaning
\a	Alert (bell)
\b	Backspace
\n	New line
\r	Carriage return
\t	Tab

The idea of using backslashes to describe escape sequences comes from the C language and has been adopted by many other programming languages, including Shell. The -e option for echo can interpret escape sequences. You can also place it within $' ' to achieve similar results. The following example uses the sleep command, which waits for a specified amount of time (in seconds) before exiting, creating a simple countdown timer:

[root@lht ~]# sleep 10; echo -e "Time's up\a"
Time's up

Alternatively, you can do it this way:

[root@lht ~]# sleep 10; echo "Time's up" $'\a'
Time's up