1 of 14

Basic Unix

Why Unix?

UNIX is an operating system (suite of programs), originally developed in 1969 at Bell Labs and has been under development ever since. UNIX is a very popular operating system for many reasons, including:

multi user (multiple users can log in to a computer at the same time, and use concurrently the resources of that computer).
multi-tasking (each user can perform many tasks at the same time).
network-ready (built-in TCP /IP networking makes easy to communicate between computers).
very powerful programming environments (free of the many limits imposed by other operating systems).
robust and stable.
scalable, portable, flexible.
open source.

UNIX systems also have a graphical user interface (GUI) similar to Microsoft Windows which provides an easy to use environment. However, knowledge of UNIX is required for operations which aren't covered by a graphical program, or for when there is no windows interface available (for example, in a telnet session).

UNIX at a glance

UNIX consists of three main components: kernel, shell, and programs.

KERNEL: hub of the operating system.
SHELL: command line interpreter.
PROGRAMS: collections of operations.

Each component has specific roles:

KERNEL: it allocates time and memory to programs and handles the filestore and communications in response to system calls.
SHELL: it interprets the commands the user types in and arranges for them to be carried out.
PROGRAM: execute a set of predefined operations.

As an illustration of the way the shell and the kernel work together, suppose a user types the following command:

rm myfile

Internally,

The shell searches the PATH environment variable for the file containing the program rm.
The shell requests to the kernel, through system calls, to execute the program rm on myfile.
When the process has finished running, the shell returns the UNIX prompt to the user, indicating that it is waiting for further commands.

The Unix Tree

The UNIX file system consists of files and directories organized in a hierarchical structure.
Avoid File names and directory names with spaces and special characters
In the figure below, there are 5 subdirectories under the root, i.e. bin, tmp, home, use, src. The directory home has two subdirectories (i.e. iap and paolaf). Thus the parent directory of iap is home.

The system provides a few short synonyms relative to the working directory:

The root is denoted by a forward slash (/).
The home directory of any given user is specified by a tilde (~).
A single dot (".") indicates the current working directory.
A double dot ("..") indicates the directory above the current directory.

You can find out your current working directory (in other words, where you currently are in the Unix tree) by typing the command pwd (print working directory) at the prompt and then hitting return.

# print working directory  
pwd

# change directory to root directory
cd /
pwd

# change directory to home directory
cd ~
pwd

# change directory to root directory
cd /
pwd

The Unix Terminal and Shell

Terminal

Common name for the interface that allows you to input commands and see the output from commands.

Command prompt

Sequence of one or more characters, such as $ or % in combination with other customizable elements, such as the username or the current working directory.
It indicates readiness to accept commands.

Shell

Command line interpreter.
Users operate the computer by entering command input as text for a shell to execute or by creating text scripts of one or more such commands.

Anatomy of a Unix Command

A basic UNIX command has 3 parts. These are:

command name
options (zero or more)
arguments (zero or more)

In general, UNIX commands are written as single strings followed by the return key. Note that UNIX commands are case-sensitive, i.e. it does matter whether you type a letter in uppercase or lowercase.

Example

wc –l myfile.txt

The example above has the following components:

the command name (wc)
one option (-l)
one argument (myfile.txt)

Options

Also called “switches” or “flags”.
Specify the way a given commands must work.
Preceded by one or two dashes (one dash if the switch is a single character, two dashes otherwise).

Tab completion

Tab completion is a useful features of UNIX where the shell automatically fills in partially typed commands when the tab key is used. This is advantageous in many ways:

Commands or filenames with long or difficult spelling require fewer keystroke to reach
In the case of multiple possible completions, the shell will list all filenames beginning with those few characters.

History

Another useful feature is the fact that the shell remembers each typed command, so you can easily recall and run previous commands.

By using the up and down arrows at the prompt, you can revisit the command history and recall previously typed commands.
Previous commands can be re-executed as they are or can be edited using the left/right arrow.

Use the history command to specify how many and which previous commands to display.

# display last typed commands
history

#display the last n typed commands
history n

#execute command n
!n

#execute last command in the list
!!

# recall and execute the n-th last command
!-n

Examples:

# recall the last 5 typed commands
history 5

# recall and execute the 3rd last command
!-3

# recall and execute command number 120
!120

Basic Unix Commands

pwd

This command name stands for "print working directory".
It displays on the terminal the absolute path name of the current (working) directory.
It is useful to locate where you are currently in the Unix tree.

# print working directory
pwd

This command name stands for "change directory".
It changes your current working directory to the specified location.
Your home directory is referred to as "~" (tilde).
The current directory is referred to with a single dot ( ".").
The directory located one level up the current directory (also known as "parent directory") is referred to with two dots ("..").
The last visited directory is referred to with an hyphen ("-").

# go to root directory of the system and print the working directory
cd /
pwd

# go to the home directory and print the working directory
cd ~
pwd

# change directory using the absolute path and print the working directory
cd /net/bmc-pub14/data/
pwd

This command stands for "list".
It displays the content of a directory.
By default, the content is listed in lexicographic order.

# list the content of the current directory
ls

# list content of the parent directory
ls ../

Command switches allows to list the directory's content with more or less detail and sorted according to different criteria.
- The switch "a" lists all files and directories, even those starting with "." that are normally hidden.
- The switch "l" lists the content in "long" format, including the total size (in blocks), mode (permission), the number of links, the owner, the size, the date of last modification, and the file or subdirectory name. Note that the first hyphen is replaced with a “d” if the item is a directory.
- The switch "t" sorts the content of the directory by time modified (most recently modified first).
- The switch "S" sorts the content of the directory by size (smaller first).
- The switch "r" reverses the order of the sort.
- The switch "R" recursively lists subdirectories encountered.
- The switch "u" specifies the use of time of last access of the file, instead of last modification, for sorting the content.
- The switch "U" specifies the use of time of file creation, instead of last modification, for sorting the content.
- The switch "F" displays a slash ("/") immediately after a directory, an asterisk ("*") after an executable, an at sign ("@") after a symbolic link.

# list all files
ls -a

# list in long format
ls -l

# list in long format, sorting by date
ls -lt

# list in reverse lexicographic order
ls -r

# list by size 
ls -S

# list in long format according to last access
ls -lu

# list in long format according to the time of creation
ls -lU

# display “/” after a directory, “*” after an executable, “@” after a symbolic link
ls -F

To view the commands available on luria, execute the following (note the space-separated list of directories to list):

ls /bin/ /usr/bin/ /usr/local/bin/

mkdir

This command name stands for "make a directory".
It creates a new folder (or directory). If no path is specified, the new directory is created in the current directory.
The switch "-v" specifies a verbose mode: a message with the folder(s) created is printed on the screen.

# create a directory named "testdir1" with a subdirectory named "testdir2"
mkdir testdir1
mkdir testdir1/testdir2

# change current directory directly to "testdir2"
cd testdir1/testdir2

# go to the parent directory (i.e. testdir1) and print the working directory
cd ..
pwd

# create a new directory named "testdir3" with the verbose mode on
mkdir -v testdir3

This command name stands for "copy".
It makes copies of files and directories to the specified location.
The switch "v" enables the verbose mode: messages describing the files copied are displayed on the screen.
The switch "i" enables the interactive mode: confirmation is requested before overwriting files.
Wildcards symbols such as "*" or "?" are commonly used to copy multiple files with a single command.
- The symbol "*" stands for any number of alphanumeric characters.
- The symbol "?" stands for a single alphanumeric character.

The following examples assume you have a directory named "unix_class" in your home directory:

# create a directory "unix_class" in your home directory and access it 
mkdir ~/unix_class
cd ~/unix_class

# copy the file named arrayDat.txt into your unix_class directory
cp /net/ostrom/data/dropbox/arrayDat.txt ~/unix_class/
ls

# the above command is equivalent to the following: 
cp /net/ostrom/data/dropbox/arrayDat.txt ./
ls

# use the interactive mode and type "y" to confirm the choice 
cp -i /net/ostrom/data/dropbox/arrayDat.txt ~/unix_class/
ls
# use the verbose mode to see the file being copied 
cp -v /net/ostrom/data/dropbox/arrayDat.txt ~/unix_class/

# make a local copy of the file call it "arrayDat1.txt" 
cp arrayDat.txt arrayDat1.txt
ls

# copy all the files with suffix "array”  into the current directory 
cp /net/ostrom/data/dropbox/UNIX/array* ./
ls

# copy any file whose extension is "txt" 
cp -i /net/ostrom/data/dropbox/UNIX/*.txt ./
ls

This command name stands for "move".
It moves (renames) files and directories.
Several switches are available to specify the behavior of this command, including "-i" (interactive mode) and "-v" (verbose).

# cp arrayDat.txt into arrayDat1.txt, then rename arrayDat1.txt as arrayDat2.txt 
cp arrayDat.txt arrayDat1.txt
mv arrayDat1.txt arrayDat2.txt
ls

# rename in interactive mode, i.e. ask for confirmation before overwriting existing file 
cp arrayDat2.txt arrayDat3.txt
ls
mv -i arrayDat2.txt arrayDat3.txt
ls

# rename in verbose mode, i.e. print information on the screen 
mv -v arrayDat3.txt arrayDat4.txt
ls

rmdir

This command stands for "remove directory".
It deletes the specified directory.
The switch "-v" specifies a verbose mode: a message with the folder(s) deleted is printed on the screen.
Note that in general you cannot delete a directory that is not empty. The content of such directory has to be deleted before the directory itself can be deleted.

# remove the folder "testdir2" in the location specified by the path
rmdir ~/testdir1/testdir2

# remove the folder "newdir1"
rmdir testdir1

# remove the folder "newdir3" using the verbose mode
rmdir -v testdir3

This command name stands for "remove".
It deletes files and directories.
Several switches are available to specify the behavior of this command, including "-i" (interactive mode) and "-v" (verbose).

# create copies of arrayDat.txt 
cp arrayDat.txt arrayDat1.txt
cp arrayDat.txt arrayDat2.txt
cp arrayDat.txt arrayDat3.txt
cp arrayDat.txt arrayDat4.txt
ls

# delete file 
rm arrayDat2.txt

# delete file in interactive mode, i.e. ask for confirmation before removing 
rm -i arrayDat3.txt

# delete file in verbose mode, i.e. prints the name of file deleted 
rm -v arrayDat4.txt

The "rm" command can be used in conjunction with wildcard symbols to delete multiple files at once. Extreme caution should be used, as action are not often reveresible. It is good practice either to use the interactive/verbose mode or use the command "ls" with the intended pattern, before invoking the command "rm".

# create a copy of arrayDat1.txt, then delete files with suffix “arrayDat”,
  followed by a digit and extension “txt”
cp arrayDat1.txt arrayDat2.txt
rm -i arrayDat?.txt

The switch "-r" deletes the content of a directory recursively,

i.e. deletes the directory's files and subdirectories, including their content.

#create a subdirectory and copy *.txt files 
mkdir testdir
cp *.txt testdir/
ls 
ls testdir/*

# remove directory and its content 
rmdir testdir
rm -r testdir
ls

cat

This command name stands for "concatenate".
It concatenates the content of file(s) and prints it out.
The switch "-n" specifies to number each line, starting from 1.
The "cat" command is often used in conjuction with the pipe ("|") to view the content of long files.
The "cat" command can be used to copy and append the content of files using output redirection (">").

# view the content of myfile.txt 
cat arrayDat.txt

# numbers each line of myfile.txt 
cat -n arrayDat.txt

# copy the content of myfile.txt using output redirection
cat arrayDat.txt > file1.txt
cat arrayDat.txt > file2.txt
ls

# concatenate the content of the two files
cat file1.txt file2.txt

# append the content of file2.txt to file1.txt
cat file2.txt >> file1.txt
cat file1.txt

When used without argument, the "cat" command takes STDIN as default argument. You can use this behavior to type in brief information, rather than invoking a standard text editor.

# type the following command 
cat > testfile.txt
# type any text, then hit Return, then press Ctrl-C 
#view content of newly created file
cat testfile.txt

less

This command is a "pager" that allows the user to view (but not modify) the contents of a text file one screen at a time.
The space-bar is used to advance to the next page.
The key "q" is used to "quit" the pager.
The switch "-n" allows to view the content of a file starting at the specified line.

# view the content of arrayDat.txt, hit the space bar to advance, type "q" to quit
less arrayDat.txt

# view content of file starting at line 5
less +5 arrayDat.txt

head

This command displays the top part of a file.
By default, the first 10 lines of a file are displayed.
The switch "-n" specifies the number of lines to display.
The switch "-b" specifies the number of bytes to display.

# display the first lines of a file (10 lines by default)
head arrayDat.txt

#display the first 2 lines of a file
head -n 2 arrayDat.txt

tail

This command displays the bottom part of a file.
By default, the last 10 lines of a file are displayed.
The switch "-n" specifies the number of lines to display.
The switch "-b" specifies the number of bytes to display.

# display the last lines of a file (10 lines by default)
tail arrayDat.txt

#display the last 2 lines of a file
tail-n 2 arrayDat.txt

top

This command displays Linux tasks
The top program provides a dynamic real-time view of a running system

This command stands for Disk Usage
It estimates file space usage

# print sizes in human readable format
du -h

This command stands for Disk Free
It reports file system disk space usage

# print sizes in human readable format
df -h

Output Redirection and Piping

Output Redirection

Most processes initiated by UNIX commands take their input from the standard input (i.e. the keyboard) and write their output to the standard output (i.e. the terminal screen).
The "<" sign can be used to redirect the input, that is to specify that the input comes from something other than the keyboard.
The ">" sign can be used to redirect the output, that is to specify that the output goes to something other than the terminal screen.
The ">>" sign can be used to append the output to something other than the terminal screen.

Piping

A pipe is denoted by "|".
Several pipes can be used in the same command line to create a "pipeline".
A pipe takes the output of a command and immediately sends it as input to another command.
A pipe is often used in conjunction with the command "less" to view the output within the pager.

Manual Pages

Unix comes with a preloaded documentation, also known as "man pages".

Anatomy of a man page

Each page is a self-contained document consisting of the following sections:

NAME: the name and a brief description of the command.
SYNOPSIS: how to use the command, including a listing of the various options and arguments you can use with the command. Square brackets ([ ]) are often used to indicate optional arguments. Any arguments or options that are not in square brackets are required.
DESCRIPTION: a more detailed description of the command including descriptions of each option.
SEE ALSO: References to other man pages that may be helpful in understanding how to use the command in question.

View man pages

To view a manual page, use the command man.

man command_name

When viewing a man page, use the following keys to navigate the pages:

spacebar- view the next screen;
b (for "back") - view the previous screen;
arrow keys - navigate within the pages;
q (for "quit") - quit and return to the shell prompt.

For example, to view the man page associated with the command cp, type the following:

man cp

Search man pages

When you are not sure of the exact name of a command, you can use the apropos command to see all the commands with the given keyword on their man page.

apropos keyword

For example, to see which commands are relevant to the task of copying, type the following:

apropos copy

Access Rights

Every user has a unique username and is member of 1+ groups.
Every file and directory has a owner, a group and a set of permission flags.
Flags specify permissions:
- read, write and execute(rwx)
- owner, group and others (ugo).
Programs need to have execute (x) permission to run in shell

Permissions are controlled with the commands chmod, chown, chgrp

Only owner can change permissions of a file

chmod

chmod has 2 different syntaxes

Syntax 1

assign (=), gives(+), or take away(-) permission
who corresponds to --> u (user), g (group), o (other)
permission corresponds to --> read (r), write (w), execute (x)
- chmod who=permission
- chmod who+permission
- chmod who-permission

chmod n

numeric shortcuts to change mode in a compact way.
format: chmod n filename
- n is a 3 digit number, values from 0 to 7
- each digit is associated to user, group, other

examples:

Unix Text Editors

nano

Nano is freely available under the GPL for Unix and Unix-like systems.
It is a keyboard-oriented editor, controlled with key combinations, i.e. the "control" key (denoted by "^") PLUS certain letter keys.

Basic commands:

^O : saves the current file
^W : goes to the search menu
^G : gets the help screen
^X : exits

Basic Usage

Launch the editor by typing "nano" at the shell prompt. An overview of available commands is given at any time at the bottom of the editor screen.

nano

Open an existing file by typing "nano" followed by the file name.

nano file1.txt

If you modify the file, when you exit you will be asked whether to save the changes and the name of the file to save.

Advanced Usage

Find and replace in nano

While holding down Ctrl key, press \
Enter your search string at prompt and press return
Enter your replacement string at prompt and press return
Respond to "Replace this instance" option menu:
- Y to replace current instance
- N to skip current instance
- A to replace all instances
Note: The search string can also be a regular expression.

vi / vim

How it works

The Unix command vi starts up the visual editor.
Typing vi followed by a file name will automatically open the file.
After issuing the command, the appearance of your screen changes. Rather than seeing the shell prompt, the content of the file filename.txt appears on the screen. If filename.txt hadn't existed before you invoked the vi command, the screen will appear mostly blank.

vi filename.txt

Modes

One of the fundamental concept to keep in mind when using vi is that three modes exist: command, insert, and visual.
In command mode, everything you type on the keyboard gets interpreted as a command.
In insert mode, most everything you type on the keyboard gets interpreted as characters and will appear in your file--letters, numbers, punctuation, line returns, etc.
When you are in insert mode, you can switch to command mode by pressing the Esc key on your keyboard (this is often at the left and upper portion of your keyboard).
When you are in command mode, there are many keys you can use to get into insert mode, each one gives you a slightly different way of starting to type your text. For example, if you are in command mode, you can simply type i on your keyboard to enter insert mode (i stands for insert). Then, the characters you type go into the file.

Basic commands

h = move one character to the left
l = move one character to the right
k = move up one line
j = move down one line
[ctrl] b = move back one screen
[ctrl] f = move forward one screen
quit Vi without saving anything (you'll lose any changes you made when using this command) type: :q!
save/write the file you're working on without exiting type: :w followed by a filename
save/write your file and quit the vi editor in one step by typing: :wq

emacs

Prior to starting this demonstration, you should set up your unix directory according to the instructions in Unix utilities
emacs is a powerful text editor with extensive functionality.

1) Control mode is activated by pressing and holding the <ctrl> key while pressing the second key. This process is written as:

C-key

This example shows the result after C-s (search):

2) Meta-mode is activated by pressing and releasing the <ESC> key followed by some other key that activates a sub-menu of commands. This process is written as:

ESC option

This example is what appears after entering ESC x (activate execute-extended-command menu)

check to see that you are in the IAP_2010 directory unix by entering the command:

pwd

the result should be (where USERNAME is your own unix username)

/home/USERNAME/IAP_2010/unix

If it is not, you can change to that directory by executing:

cd /home/USERNAME/IAP_2010/unix

create a copy of the file replace.txt with:

cp replace.txt replace2.txt

Begin editing replace2.txt with:

emacs replace2.txt

Note, on new accounts a splash screen welcomes you to emacs use C-l to exit the screen and proceed to editing.
Insert text by regular typing.
Delete letters with C-d
Delete lines with C-k
Find and replace text with ESC x , then type replace-string on M-x line. enter search string, then return, then replacement text.
Move to the end of the document with ESC >
Move back to the start of the document with ESC <
Play tetris with ESC x, tetris
Save and exit with C-x followed by C-c then answer prompts if changes were made.
To turn off the splash welcome screen, edit a file called .emacs to have the contents:

(setq inhibit-splash-screen t)

Shell Scripts

Shell scripts are sequential collection of commands that generally represent a workflow.
Shell scripts can use other commands as building blocks.