Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
scp
Allows you to transfer files between a local and remote host. It does this by leveraging ssh.
Shares the same syntax as the normal cp command.
# Transfer from local computer to remote computer
scp <source file> <username>@<remote host>:<destination>
scp agenda.txt [email protected]:/home/asoberan/unxiclass
# Transfer from remote computer to local computer
scp <username>@<remote host>:<source file> <destination>
scp [email protected]:/home/asoberan/arrayDat.txt .nano
A basic text editor found on most UNIX operating systems.
Gives instructions on how to use it at the bottom of the screen.
^ stands for Ctrl
Example output from running nano arrayDat.txt. Save any changes by exiting with ^X:
GNU nano 2.3.1 File: arrayDat.txt
ProbeID Sample1 Sample2 Sample3 Sample4
1007_s_at 10.93 11.44 11.19 11.64
1053_at 8.28 7.54 8.06 7.32
117_at 3.31 3.41 3.13 3.13
121_at 4.42 4.32 4.46 4.63
1255_g_at 1.8 1.7 1.75 1.81
^G Get Help ^O WriteOut ^R Read File ^Y Prev Page ^K Cut Text ^C Cur Pos
^X Exit ^J Justify ^W Where Is ^V Next Page ^U UnCut Text ^T To SpellIf you recall, you can mount the storage servers to your local machine using SMB. Since mounting the server makes it look like a local directory on your computer, you can use whatever text editing software that you're comfortable with to edit text files on the computing cluster.
Until now, all the examples of using a shell that we've seen have used our keyboard input as STDIN and the terminal screen as STDOUT. However, the shell gives us the ability to change STDIN and STDOUT by using two mechanisms: pipes and redirectors.
A redirector is denoted by a >. A redirector redirects the output of a command into a file, essentially changing the STDOUT to a file rather than the terminal screen.
For example, echo 'Hello, World!' usually outputs the message "Hello, World!" onto our terminal screen. However, if we run
nothing will be printed to the screen. Instead, STDOUT is changed to the file hello.txt, so the output of echo 'Hello, World!' will be pasted into the file hello.txt.
We can verify that by running cat hello.txt, which should print out the contents of hello.txt to our terminal screen.
A redirector will always replace the contents of a file. If, however, you wish to append content to a file, you can use >>.
Using >> to append contents to a file:
A pipe is denoted by a |. Pipes allow us to use the output of a command as the input of another command.
For example, let's say you wanted to search through the contents of cities.csv to search for every line that includes "TX". You can do so by printing the contents of the file using cat, then piping the output into grep "TX".
This will print every line that includes "TX".
Piping is a very powerful feature of the shell, because it lets us compose many small operations into a very complex set of operations. For example:
echo 'Hello, World!' > hello.txtecho 'Hello, again!' >> hello.txtcat cities.csv | grep "TX"# List the files in the current directory
# and print extended information
ls -l# Format the output of ls -l by removing
# cases of multiple spaces and replace
# them with a single space for easier
# parsing
ls -l | tr -s " "# Use the cut program to separate each line
# of output into colums, delimited by a space
# then print columns 3-4. In the case of this
# output, we're printing the users and groups
# which own the files in the directory
ls -l | tr -s " " | cut -d" " -f3-4# Alphabetically sort the users and groups which
# own the files in the directory
ls -l | tr -s " " | cut -d" " -f3-4 | sort# Count the instances of users and groups so we can
# know who owns files in the current directory and
# how many files they own
ls -l | tr -s " " | cut -d" " -f3-4 | sort | uniq -ccat
Concatenates two files together, then prints content to STDOUT.
Colloquially, it's used to print out the contents of a single file.
# Start in ~/unixclass
cd ~/unixclass
cat arrayDat.txthead
By default, prints the first 10 lines of a file to STDOUT.
You can pass the -n flag to specify the number of lines to print.
# Start in ~/unixclass
cd ~/unixclass
# Print the first 10 lines of arrayDat.txt
head arrayDat.txt
# Print the first 5 lines of arrayDat.txt
head -n 5 arrayDat.txttail
By default, prints the last 10 lines of a file to STDOUT.
You can pass the -n flag to specify the number of lines to print.
# Start in ~/unixclass
cd ~/unixclass
# Print the last 10 lines of arrayDat.txt
tail arrayDat.txt
# Print the last 5 lines of arrayDat.txt
tail -n 5 arrayDat.txtless
Lets you page through a long file or stream of text a.k.a. you can scroll.
Exit by pressing q
# Start in ~/unixclass
cd ~/unixclass
# Scroll through the contents of arrayDat.txt
less arrayDat.txtUNIX filesystems have a feature called symbolic links, which are files that point to another file, called the target. They're similar to shortcuts in Windows.
There are two kinds of symbolic links: hard links and soft links.
Hard links create a file which points to the same space on the hard drive as the file which is being linked to, rather than to that file itself.
A file won't be deleted until every hard link to it is deleted.
To create a hard link, you use the ln command with the source file, and the target hard linked file:
Notice that any changes you make to arrayDat.txt will be reflected in arrayHard.txt.
Soft links create a file which points to the original file or directory.
Soft links break if the original file is deleted.
To create a soft link, you use the ln command with the -s flag.
Notice that any changes you make to arrayDat.txt will be reflected in arraySoft.txt. The ls command will show that arraySoft.txt points to the file arrayDat.txt.
Your Luria home folder has a couple of soft links automatically set up pointing to storage servers so that common programs don't take up too much space on the head node.
The following sections explain how to view, create, and edit files all through the Unix shell. While it is possible to do these things straight from the shell, if you are using Visual Studio Code set up as described on the page linked below, you can view, create, and edit files through it instead of the shell.
To change the owners of a file, you can use the following commands:
chown
This changes the user who owns a particular file or directory.
chgrp
This changes the group who owns a particular file or directory.
To change the permissions that the owners of a file have, you use the chmod command.
chmod takes two arguments: the permissions to give a file, and the file to change the permissions of. The permissions are represented as a 3-digit number, where each digit represents the permissions to give the user, group, or others, respectively.
Read, write, and execute permissions are represented by the following numbers:
r - 4
w - 2
x - 1
If you want to give someone multiple permissions, you add the numeric representations of those permissions together. For example:
Read, write, execute (rwx) permissions = (4 + 2 + 1) = 7
Write, execute (_wx) permissions = (2 + 1) = 3
So let's say you want to give a file the following permissions:
The user that owns the file should be able to read, write, and execute the file. rwx = (4 + 2 + 1) = 7
The group that owns the file should be able to read and execute the file. r_x = (4 + 1) = 5
Anyone else should have no permissions for the file. ___ = 0
The you'd run the following command:
chmod 750 arrayDat.txtRemembering the syntax for this command can be quite cumbersome, so I recommend using a third-party website such as https://quickref.me/chmod.
/home/me/file /home/you/file
| |
hard link hard link
| ,'
V _.-`
HARD DRIVE <....--``ln arrayDat.txt arrayHard.txt
ls/home/me/file <---Soft link--- /home/you/file
|
|
V
HARD DRIVEln -s arrayDat.txt arraySoft.txt
ls -lquota
Luria has a disk usage quota on the head node. You can use the quota command to check what the size of the quota is and how much space you're currently using.
The -s flag will display the quota in a human-readable format. (e.g. 18K, 14M, 65G).
du
Stands for "disk usage". Reports how much disk space a directory is taken up.
Defaults to displaying the disk usage in kilobytes, but passing it the -h flag will return disk usage in a human-readable format. (e.g. 18K, 14M, 65G).
Will search the entire depth of the Unix tree starting from the directory you give it. You can control the depth which it searches by passing the -d <num> flag.
tar
Combines multiple files or directories into one archive for easy sharing. Similar to "zipping" files, however tar does not compress by default.
Create an archive by passing the -cf flags
Can compress multiple files/folders using gzip by passing the -z flag when creating a new archive.
Useful when you have files that take up a lot of space and you want to save space.
Extract an archive by passing the -xf flags. Un-compress an archive by passing the -z flag to those two flags.
zip
Zip multiple files and directories into one file. Zipping is similar to archiving with tar, but zipped files are easier to deal with on Windows machines.
quota -s
Disk quotas for user asoberan (uid 247789):
Filesystem space quota limit grace files quota limit grace
/dev/mapper/cl-home
4391M 18432M 100G 24925 0 0
# Space column shows how much space you're currently using on the head node
# Quota column is total amount of space allotted to you# Displays the disk usage of every file under your home directory
# The last entry will be the disk usage of your entire home directory
du -h ~
# Displays the disk usage of every file and directory immediately
# under your home directory
du -h -d 1 ~
# Displays the disk usage of one file, file.txt
du -h file.txt# Create an archive of a directory and name it my_directory.tar
tar -cf my_directory.tar <directory>
# Create a compressed archive of a directory and
# name it my_compressed_directory.tar.gz
tar -czf my_compressed_directory.tar.gz <directory>
# Extract archive my_directory.tar
tar -xf my_directory.tar
# Uncompress the archive my_compressed_directory.tar.gz
tar -xzf my_compressed_directory.tar.gz# Zip a directory to the file my_directory.zip
zip my_directory.zip <directory>
# Unzip my_directory.zip
unzip my_directory.zipUNIX is a multi-user environment, how does it maintain security inside of itself?
Every file has an owner and permissions.
There are three levels of ownership:
User
Group
Other
Three levels of permissions:
Read
Write
Execute
How is this useful? Well imagine a lab! There are files that an entire lab should have access to. So put all users in a lab into a lab group, then sharing a file between a lab just means making the lab group the owner of a file. This is already what we do on Luria!
You can view the ownership and permissions of a file by running ls -l. Here's an example of the output of ls -l:
[asoberan@luria unixclass]$ ls -l
total 40
-rwxr-xr-x 1 asoberan ki-bcc 3845 Apr 28 21:48 arrayAnnot.txt
-rwxr-xr-x 2 asoberan ki-bcc 3134 Apr 28 22:11 arrayDat.txt
-rwxr-xr-x 2 asoberan ki-bcc 3134 Apr 28 22:11 arrayHard.txt
-rwxr-xr-x 1 asoberan ki-bcc 1634 Apr 28 21:48 arraylen.txt
lrwxrwxrwx 1 asoberan ki-bcc 12 Apr 28 22:13 arraySoft.txt -> arrayDat.txt
-rwxr-xr-x 1 asoberan ki-bcc 3128 Apr 28 21:48 beep.txt
-rw-r--r-- 1 asoberan ki-bcc 528 Apr 28 21:48 ex1.sh
-rw-r--r-- 1 asoberan ki-bcc 479 Apr 28 21:48 ex2.sh
-rw-r--r-- 1 asoberan ki-bcc 368 Apr 28 21:48 ex3.sh
-rwxr-xr-- 1 asoberan ki-bcc 340 Apr 28 21:48 test_1.fastq
-rwxr-xr-- 1 asoberan ki-bcc 340 Apr 28 21:48 test_2.fastqLet's focus on the arrayDat.txt file.
-rwxr-xr-x 2 asoberan ki-bcc 3134 Apr 28 22:11 arrayDat.txtasoberan ki-bcc describes the ownership of a file. In this case, the user asoberan and the group ki-bcc own the file.
-rwxr-xr-x describes the permissions that the owners of the file have.
The permissions can be broken down into three parts:
The user's permissions
-rwx
The user asoberan has read (r), write (w), and execute (x) permissions for this file.
The group's permissions
r-x
The group ki-bcc has read (r) and execute (x) permissions for this file.
Everyone's else's permissions
r-x
Anyone who isn't asoberan or in the group ki-bcc has read (r) and execute (x) permissions for this file.
To check what group you are in, you can use the groups command:
[asoberan@luria unixclass]$ groups
ki-bcc