Often, I use a Linux shell script that executes several commands. Sometimes you don’t want to wait for a command to end and you want the next command to immediately run after the previous one is started.
In my previous article I mentioned such a situation, with regard to preparing a minikube dashboard.
[https://technology.amis.nl/continuous-delivery/containers/adding-grafana-to-my-vm-with-minikube-and-elasticsearch/]
Remember, the content of my minikube.sh script looks like:
#!/bin/bash echo "**** Begin downloading minikube" #Kubernetes 1.22.3 requires conntrack to be installed in root's path sudo apt install -y conntrack #Download a static binary curl -o minikube https://storage.googleapis.com/minikube/releases/v1.24.0/minikube-linux-amd64 chmod +x minikube #Add the Minikube executable to your path sudo cp minikube /usr/local/bin/ rm minikube echo "**** End downloading minikube" echo "**** Begin starting a Cluster" sudo sysctl fs.protected_regular=0 #Start a Cluster minikube start \ --vm-driver=none \ --extra-config=kubeadm.node-name=minikube \ --extra-config=kubelet.hostname-override=minikube #To use kubectl or minikube commands as your own user, you may need to relocate them. sudo cp -R /root/.kube /root/.minikube /home/vagrant sudo chown -R vagrant /home/vagrant/.kube /home/vagrant/.minikube sed -i 's/root/home\/vagrant/g' /home/vagrant/.kube/config echo "**** End starting a Cluster" echo "**** Begin preparing dashboard" minikube dashboard --url </dev/null &>/dev/null & echo "**** End preparing dashboard" minikube kubectl -- get pods -A
There I already explained a little bit about using a Linux background process and redirecting the standard IO streams, with regard to the command:
minikube dashboard –url </dev/null &>/dev/null &
In this article I will dive a little bit deeper into Linux foreground and background processes and also redirection, by showing you some examples.
Foreground process
When you run a command in the terminal, you have to wait until the command finishes before you can enter another one. This is called running the command in the foreground or foreground process. When a process runs in the foreground, it occupies your shell, and you can interact with it using the input devices.
Background process
If a command takes a long time before it finishes, and you want to start another command in the meantime, one of the options you have, is to run the command in the background. Another option is to start a new shell session and run the command in it.
A background process is a process/command that is started from a terminal and runs in the background, without interaction from the user. The shell prompt is displayed immediately after you press Return.
To run a command in the background, add the ampersand symbol (&) at the end of the command.
The shell job ID (surrounded with brackets) and process ID will be printed on the terminal.
Streams
Under normal circumstances every UNIX program has three streams opened for it when it starts up, one for input, one for output, and one for printing diagnostic or error messages. These are typically attached to the user’s terminal but might instead refer to files or other devices, depending on what the parent process chose to set up.
The input stream is referred to as “standard input”; the output stream is referred to as “standard output”; and the error stream is referred to as “standard error”. These terms are abbreviated to form the symbols used to refer to these files, namely stdin, stdout, and stderr.
[https://man7.org/linux/man-pages/man3/stdin.3.html]
Standard input (stdin)
Standard input is a stream from which a program reads its input data. The program requests data transfers by use of the read operation. Not all programs require stream input. For example, the dir and ls programs (which display file names contained in a directory) may take command-line arguments, but perform their operations without any stream data input.
Unless redirected, standard input is inherited from the parent process. In the case of an interactive shell, that is usually associated with the keyboard.
The file descriptor for standard input is 0 (zero).
[https://en.wikipedia.org/wiki/Standard_streams]
Standard output (stdout)
Standard output is a stream to which a program writes its output data. The program requests data transfer with the write operation. Not all programs generate output. For example, the file rename command (variously called mv, move, or ren) is silent on success.
Unless redirected, standard output is inherited from the parent process. In the case of an interactive shell, that is usually the text terminal which initiated the program.
The file descriptor for standard output is 1 (one).
[https://en.wikipedia.org/wiki/Standard_streams]
Standard error (stderr)
Standard error is another output stream typically used by programs to output error messages or diagnostics. It is a stream independent of standard output and can be redirected separately.
This solves the semi-predicate problem, allowing output and errors to be distinguished, and is analogous to a function returning a pair of values – see Semi-predicate problem: Multi valued return. The usual destination is the text terminal which started the program to provide the best chance of being seen even if standard output is redirected (so not readily observed). For example, output of a program in a pipeline is redirected to input of the next program or a text file, but errors from each program still go directly to the text terminal so they can be reviewed by the user in real time.
It is acceptable and normal to direct standard output and standard error to the same destination, such as the text terminal. Messages appear in the same order as the program writes them, unless buffering is involved. For example, in common situations the standard error stream is unbuffered but the standard output stream is line-buffered; in this case, text written to standard error later may appear on the terminal earlier, if the standard output stream buffer is not yet full.
The file descriptor for standard error is defined by POSIX as 2 (two).
[https://en.wikipedia.org/wiki/Standard_streams]
Redirection
Before a command is executed, its standard input (file descriptor 0), standard output (file descriptor 1) and standard error output (file descriptor 2) may be redirected using a special notation interpreted by the shell.
Redirection allows commands’ file handles to be duplicated, opened, closed, made to refer to different files, and can change the files the command reads from and writes to. Redirection may also be used to modify file handles in the current shell execution environment. The following redirection operators may precede or appear anywhere within a simple command or may follow a command. Redirections are processed in the order they appear, from left to right.
Redirection of input causes the file whose name results from the expansion of word to be opened for reading on file descriptor n, or the standard input (file descriptor 0) if n is not specified.
Redirection of output causes the file whose name results from the expansion of word to be opened for writing on file descriptor n, or the standard output (file descriptor 1) if n is not specified. If the file does not exist it is created; if it does exist it is truncated to zero size.
The following construct allows both the standard output (file descriptor 1) and the standard error output (file descriptor 2) to be redirected to the file whose name is the expansion of word.
[https://www.gnu.org/software/bash/manual/html_node/Redirections.html]
Example foreground process
So, let’s start with some examples.
First, I created a shell script (named counter.sh), with the following content:
#!/bin/bash echo "**** Begin counter" for ((i=1;i<=5;i++)); do sleep 2s echo $i echo "-----------------------------------------" done echo "**** End counter"
This script is a simple counter.
I opened a Linux Command Prompt where I used the following command:
./counter.sh
With the following output:
**** Begin counter 1 ----------------------------------------- 2 ----------------------------------------- 3 ----------------------------------------- 4 ----------------------------------------- 5 ----------------------------------------- **** End counter
Let’s start it again and this time, also issue another command, for example ps right after the first command.
./counter.sh ps
With the following output:
**** Begin counter ps 1 ----------------------------------------- 2 ----------------------------------------- 3 ----------------------------------------- 4 ----------------------------------------- 5 ----------------------------------------- **** End counter ps PID TTY TIME CMD 39985 pts/0 00:00:00 bash 44674 pts/0 00:00:00 ps
We can see the ps command is visible right after the counter started, but the command is executed after the counter (as a foreground job) has finished.
Example background process
Let’s start the counter as a background process, and again, also issue the ps command right after the first command:
./counter.sh & ps
With the following output:
[1] 46737 **** Begin counter 1 ----------------------------------------- 2 ----------------------------------------- ps PID TTY TIME CMD 46353 pts/0 00:00:00 bash 46737 pts/0 00:00:00 counter.sh 46744 pts/0 00:00:00 sleep 46745 pts/0 00:00:00 ps 3 ----------------------------------------- 4 ----------------------------------------- 5 ----------------------------------------- **** End counter ^C [1]+ Done ./counter.sh
So, this time the output from the ps command is shown immediately. We can also see the shell job ID (surrounded with brackets) and process ID printed on the terminal.
The job ID is 1 and the process ID is 46737.
I issued a Ctrl-C to stop the background job.
With the jobs command you display the status of jobs in the current session.
Start the counter as a background process, and also issue the jobs command right after the first command:
./counter.sh & jobs
With the following output:
[1] 54302 **** Begin counter 1 ----------------------------------------- jobs [1]+ Running ./counter.sh & 2 ----------------------------------------- 3 ----------------------------------------- 4 ----------------------------------------- 5 ----------------------------------------- **** End counter ^C [1]+ Done ./counter.sh
The job ID is 1 and the process ID is 54302.
The output from the jobs command is shown immediately. As you can see the background job belonging to our command is running.
I issued a Ctrl-C to stop the background job.
For more information about jobs, background processes, etc. please see the Internet.
For example: https://www.baeldung.com/linux/foreground-background-process
Example input redirection
Next, I changed the content of the counter shell script to:
#!/bin/bash echo "**** Begin counter" echo "**** Enter value for counter: " read counter echo "-----------------------------------------" for ((i=1;i<=$counter;i++)); do sleep 2s echo $i echo "-----------------------------------------" done echo "**** End counter"
Now the scripts ask’s for the counter value to use.
On the Linux Command Prompt, I used the following (foreground) command:
./counter.sh
When asked to enter a value for counter, I entered 3.
With the following output:
**** Begin counter **** Enter value for counter: 3 ----------------------------------------- 1 ----------------------------------------- 2 ----------------------------------------- 3 ----------------------------------------- **** End counter
As you can see, the counter counted to three.
In order to use input redirection, I created an input file named my_input.txt, containing the value 7, with the following (foreground) commands:
touch my_input.txt echo "7" >my_input.txt
Remark:
By using > I actually am using output redirection also.
Next, I used the (foreground) command to concatenate my file to standard output (or with other words to show the content of file my_input.txt):
[https://man7.org/linux/man-pages/man1/cat.1.html]
cat my_input.txt
With the following output:
7
Again, I changed the content of the counter shell script to:
#!/bin/bash echo "**** Begin counter" if [ -t 0 ]; then echo "Standard input coming from keyboard" echo "Enter value for counter: " read counter else echo "Standard input coming from a file or pipe" counter=$(cat /dev/stdin) echo "Received value for counter: " $counter fi echo "-----------------------------------------" for ((i=1;i<=$counter;i++)); do sleep 2s echo $i echo "-----------------------------------------" done echo "**** End counter"
This gives me the ability to start the counter script with the counter value coming from the keyboard or from a file (via input redirection).
So, let’s try both.
On the Linux Command Prompt, I used the following (foreground) command:
./counter.sh
When asked to enter a value for counter, I entered 3.
With the following output:
**** Begin counter Standard input coming from keyboard Enter value for counter: 3 ----------------------------------------- 1 ----------------------------------------- 2 ----------------------------------------- 3 ----------------------------------------- **** End counter
On the Linux Command Prompt, I used the following (foreground) command with input redirection:
./counter.sh <my_input.txt
Which is equivalent to: ./counter.sh 0<my_input.txt
With the following output:
**** Begin counter Standard input coming from a file or pipe Received value for counter: 7 ----------------------------------------- 1 ----------------------------------------- 2 ----------------------------------------- 3 ----------------------------------------- 4 ----------------------------------------- 5 ----------------------------------------- 6 ----------------------------------------- 7 ----------------------------------------- **** End counter
So, this time the value from my file was used.
Example output redirection
On the Linux Command Prompt, I used the following (foreground) command with output redirection:
./counter.sh >my_output.txt
Which is equivalent to: ./counter.sh 1>my_output.txt
As expected, I didn’t see the question where I was asked to enter a value for variable counter.
Because the command wasn’t doing anything, I entered 4 and after a few seconds the command was ready.
With the following output:
$
Next, I used the (foreground) command to show the content of file my_output.txt):
cat my_output.txt
With the following output:
**** Begin counter Standard input coming from keyboard Enter value for counter: ----------------------------------------- 1 ----------------------------------------- 2 ----------------------------------------- 3 ----------------------------------------- 4 ----------------------------------------- **** End counter
I also tried the combination of input and output redirection.
On the Linux Command Prompt, I used the following (foreground) command:
./counter.sh <my_input.txt >my_output.txt
With the following output:
$
And again, I used the (foreground) command:
cat my_output.txt
With the following output:
**** Begin counter Standard input coming from a file or pipe Received value for counter: 7 ----------------------------------------- 1 ----------------------------------------- 2 ----------------------------------------- 3 ----------------------------------------- 4 ----------------------------------------- 5 ----------------------------------------- 6 ----------------------------------------- 7 ----------------------------------------- **** End counter
Remark:
If you want to append output the existing content of a file you can use >> instead of >.
Example error redirection
Again, I changed the content of the counter shell script to:
#!/bin/bash echo "**** Begin counter" if [ -t 0 ]; then echo "Standard input coming from keyboard" echo "Enter value for counter: " read counter else echo "Standard input coming from a file or pipe" counter=$(cat /dev/stdin) echo "Received value for counter: " $counter fi echo "-----------------------------------------" for ((i=1;i<=$counter;i++)); do sleep 2s echo $i echo "-----------------------------------------" done nonexistingcommand echo "**** End counter"
I added a non-existing command in order to raise an error.
On the Linux Command Prompt, I used the following (foreground) command:
./counter.sh <my_input.txt >my_output.txt
With the following output:
./counter.sh: line 19: nonexistingcommand: command not found
As expected, the error message still is sent to the terminal display.
On the Linux Command Prompt, I used the following (foreground) command with error redirection:
./counter.sh <my_input.txt 1>my_output.txt 2>my_error.txt
Which is equivalent to: ./counter.sh 0<my_input.txt 1>my_output.txt 2>my_error.txt
This time I did have to use the file descriptor to differentiate between output (1) and error (2).
With the following output:
$
And again, I used the (foreground) command:
cat my_error.txt
With the following output:
./counter.sh: line 19: nonexistingcommand: command not found
Example both output and error redirection
On the Linux Command Prompt, I used the following (foreground) command with both output and error redirection:
./counter.sh <my_input.txt &>my_output_error.txt
With the following output:
$
And again, I used the (foreground) command:
cat my_output_error.txt
With the following output:
**** Begin counter Standard input coming from a file or pipe Received value for counter: 7 ----------------------------------------- 1 ----------------------------------------- 2 ----------------------------------------- 3 ----------------------------------------- 4 ----------------------------------------- 5 ----------------------------------------- 6 ----------------------------------------- 7 ----------------------------------------- ./counter.sh: line 19: nonexistingcommand: command not found **** End counter
Example with redirection using the null device
The null device /dev/null is a device file that discards all data written to it but reports that the write operation succeeded. The null device is typically used for disposing of unwanted output streams of a process, or as a convenient empty file for input streams. This is usually done by redirection.
[https://en.wikipedia.org/wiki/Null_device]
On the Linux Command Prompt, I used the following (foreground) command with both output and error redirection using the null device:
./counter.sh <my_input.txt &>/dev/null
With the following output:
$
So now, no files where created.
Example piping between programs
A pipeline is a mechanism for inter-process communication using message passing. A pipeline is a set of processes chained together by their standard streams, so that the output text of each process (stdout) is passed directly as input (stdin) to the next one.
The standard shell syntax for anonymous pipes is to list multiple commands, separated by vertical bars (“pipes” in common Unix verbiage):
command1 | command2 | command3
[https://en.wikipedia.org/wiki/Pipeline_(Unix)]
I created a shell script (named a program1.sh), with the following content:
#!/bin/bash echo "10"
On the Linux Command Prompt, I used the following (foreground) command with error redirection:
./program1.sh | ./counter.sh 2>my_error.txt
With the following output:
**** Begin counter Standard input coming from a file or pipe Received value for counter: 10 ----------------------------------------- 1 ----------------------------------------- 2 ----------------------------------------- 3 ----------------------------------------- 4 ----------------------------------------- 5 ----------------------------------------- 6 ----------------------------------------- 7 ----------------------------------------- 8 ----------------------------------------- 9 ----------------------------------------- 10 ----------------------------------------- **** End counter
Here you see that the standard output (stdout) of command1 (program1.sh) is passed directly as standard input (stdin) to command2 (counter.sh).
For more information about redirection, etc. please see the Internet.
For example: https://www.baeldung.com/linux/pipes-redirection
So now it’s time to conclude this article. In this article I shared with you some examples about Linux foreground and background processes and also redirection. Have fun trying them out.
