Manipulating Files

Until now, all the examples of using a shell that we've seen have used our keyboard input as STDIN and the terminal screen as STDOUT. However, the shell gives us the ability to change STDIN and STDOUT by using two mechanisms: pipes and redirectors.

Redirectors

A redirector is denoted by a >. A redirector redirects the output of a command into a file, essentially changing the STDOUT to a file rather than the terminal screen.

For example, echo 'Hello, World!' usually outputs the message "Hello, World!" onto our terminal screen. However, if we run

echo 'Hello, World!' > hello.txt

nothing will be printed to the screen. Instead, STDOUT is changed to the file hello.txt, so the output of echo 'Hello, World!' will be pasted into the file hello.txt.

We can verify that by running cat hello.txt, which should print out the contents of hello.txt to our terminal screen.

A redirector will always replace the contents of a file. If, however, you wish to append content to a file, you can use >>.

Using >> to append contents to a file:

echo 'Hello, again!' >> hello.txt

Pipes

A pipe is denoted by a |. Pipes allow us to use the output of a command as the input of another command.

For example, let's say you wanted to search through the contents of cities.csv to search for every line that includes "TX". You can do so by printing the contents of the file using cat, then piping the output into grep "TX".

cat cities.csv | grep "TX"

This will print every line that includes "TX".

Piping is a very powerful feature of the shell, because it lets us compose many small operations into a very complex set of operations. For example:

# List the files in the current directory
# and print extended information
ls -l

# Format the output of ls -l by removing 
# cases of multiple spaces and replace
# them with a single space for easier
# parsing
ls -l | tr -s " "

# Use the cut program to separate each line
# of output into colums, delimited by a space
# then print columns 3-4. In the case of this
# output, we're printing the users and groups
# which own the files in the directory
ls -l | tr -s " " | cut -d" " -f3-4

# Alphabetically sort the users and groups which
# own the files in the directory
ls -l | tr -s " " | cut -d" " -f3-4 | sort

# Count the instances of users and groups so we can
# know who owns files in the current directory and
# how many files they own
ls -l | tr -s " " | cut -d" " -f3-4 | sort | uniq -c