                   
		     	    [0;33mCloud-SPAN Genomics Course Environment 
			     		    ([0;33mCSEnv[0m)

This guide is a brief introduction to the organisation of [0;33mCSEnv[0m and its use, under these topics:

t1. [0;33mOverview of [0;33mCSEnv[0m: structure and software tools available.
t2. [0;33mIntroducing the Command Line Interface[0m ([0;33mCLI[0m): components and overall operation.
t3. [0;33mUsing [0;33mCSEnv[0m: moving around and managing files.
t4. [0;33mScripting[0m: tips and examples to help you get started.
t5. [0;33mBackup your work[0m: software and data.
t6. [0;33mSome Resources[0m: for you to go deeper into scripting.

Move around the guide by scrolling with the mouse or pressing the arrow keys or the following keys: 

[0;33mKey/s[0m     [0;33mEffect[0m
 f     	  move forward one window  — pressing the Space Bar has the same effect        
 b        move backwards one window
 g        move to the beginning (first line)
 G        move to the end (last line)
 /text[1;32m↵[0m   search for "text" and move to it — press the Enter key ([1;32m↵[0m) at the end	
 n        move to next ocurrence of "text" — after you have searched for "text" 
 N        move to previous ocurrence of "text" — after you have searched for "text" 
 h        get Help on other commands — quit the help menu by pressing the q key
 q        Quit (exit) the guide 


		 Topic 1. [37m[45mOverview of CSEnv: structure and software tools available(B[m

[0;33mCSEnv[0m is a Linux (Ubuntu 20.04) virtual machine (VM) running on Amazon Web Services (AWS). It is an instance (a copy) of the Amazon Machine Image (AMI) configured with 'omics data and software analysis tools for the Cloud-SPAN Genomics course. Participants of the course have, each, exclusive access to an instance of [0;33mCSEnv[0m for the span of the course only. Hence, you may be interested in making your own instance from that AMI image and/or backing up your software — see "Backup your your".

  As a Linux machine, [0;33mCSEnv[0m benefits from many software tools readily available, most of them free and open. But perhaps more relevant to the aim of the course: software tools in Linux can easily be combined into workflows which are the basis to automate long-run, multiple-step tasks. How to combine software tools is presented below (in Scripting). Please note that [0;32msoftware tools[0m are also known as [0;32mprograms[0m and [0;32mapplications[0m and the three terms are used interchangeably in this guide. 

  The Linux file system is structured/organised around directories (folders) which, conceptually, are files that contain files or other directories. Technically, directories contain the [0;32minformation necessary to access[0m other files or other directories. Proper files are (not directories but) documents, programs, photos, videos, etc. The highest level directory in linux is called "root" and is represented by a slash [1;33m/[0m. The root directory contains these directories and files:

[0;33mbin   dev  home[0m        [31m[40m[1minitrd.img.old(B[m  [0;33mlib32  lost+found  mnt  proc  run   snap  sys  usr[0m  [0;36mvmlinuz[0m
[0;33mboot  etc[0m  [0;36minitrd.img[0m  [0;33mlib             lib64  media       opt  root  sbin  srv[0m   [30m[42mtmp(B[m  [0;33mvar[0m  [31m[40m[1mvmlinuz.old(B[m

  Directory names are shown in [0;33mbrown[0m. Names in [0;36mcyan[0m are [0;32mlinks[0m that point to either a file or a directory — links are very useful as we'll see later. Links in [31m[40m[1mred(B[m are unresolved: the files they point to no longer exist — but don't worry, the unresolved links above became so when we updated [0;33mCSEnv[0m to Ubuntu 20.04, which no longer handles those links in the root directory but in the [0;33mboot[0m directory. The black text and green background for the [30m[42mtmp(B[m directory mean that any user including you can create and write (modify) files and directories within the [30m[42mtmp(B[m directory.

  [34m[47m NB (B[m when you create, delete or modify a file, the information about the file (name, size, etc.) in the directory that contains the file is also created, deleted or modified.  

  It is beyond the scope of this guide to even try to summarise what is in each directory in the root directory. Please visit this link for a good discussion of the role and content of each directory: 
  [33m[44m[1mhttps://www.howtogeek.com/117435/htg-explains-the-linux-directory-structure-explained/(B[m


  [0;32mYour Home Directory[0m

Within the Linux file structure, every user including you has a [0;32mhome directory[0m wherein you can create files and directories that will exist until you explicitly delete them or while your [0;33mCSEnv[0m exist. For all practical purposes, you have exclusive access to your home directory and its content (files and directories therein) but you can share some content with other users as needed.

  The name of your home directory is [0;33m/home/csuser[0m

  The name of a file in the Linux file system is unique and comprises the full location, better known as the full [0;32mpath[0m, within the file hierarchical structure which starts at the root directory [1;33m/[0m. So, your home directory is [0;33mcsuser[0m (as your login username) and is locatd within the [0;33mhome[0m directory which is in turn located within the root directory [1;33m/[0m.  

  [34m[47m NB (B[m the slash [1;33m/[0m is used both to represent the root directory, when used at the beginning of a full path, and to separate directory names within a full path. 
  [37m[1mNOTE ALSO(B[m that, in Windows systems, the inverted slash [1;33m\[0m plays only the second role of separating directory names in a full path.

  You rarely have to specify the full path of a file or directory, which is most convenient, because you will mostly use paths relative to your home directory as we will see later. 


  [0;32mGenomics-related Applications in [0;33mCSEnv[0m[0m

  As mentioned above, [0;33mCSEnv[0m benefits from many software tools readily available in Linux. We will discuss some of them in the following topics. We close this topic with a brief description of the sotware tools for the analysis of 'omics data that are readily available in [0;33mCSEnv[0m: 

SOFTWARE        VERSION		DESCRIPTION
———————————————————————————————————————————
- fastqc        0.11.9		Quality control tool for high throughput sequence data.
- trimmomatic	0.39		A flexible read trimming tool for Illumina NGS data.
- bwa 		0.7.17-r1188	Mapping DNA sequences against reference genome.
- samtools	1.13		Visualization and interactive exploration of large genomics datasets.
- bcftools	1.14-3		Utilities for variant calling and manipulating VCFs and BCFs.
- cutadapt	3.5		Similar to bcftools




	Topic 2. [37m[45mIntroducing the Command Line Interface (CLI): components and overall operation(B[m

The command line interface ([0;33mCLI[0m) is a means for you to interact with Linux: both for you to instruct Linux the operations you want to be performed, such as running a program or copying or deleting a file, and for you to visualise the results of the operations that you instruct.
    
   A [0;33mCLI[0m is made up of four main components: (1) a [0;32mtext-only[0m display or screen like the one wherein you are reading this guide; (2) the keyboard through which you enter commands (instructions) or data or both; (3) a prompt that indicates that the [0;33mCLI[0m is ready for you to enter another command; and (4) the [0;33mCLI[0m program itself that, iteratively, reads a command you enter, performs the command, displays the results on the screen if any (results may be written to a file), and again displays the prompt and waits for another command.

   This is the [0;33mCLI[0m prompt of your [0;33mCSEnv[0m instance:

[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $
 
   It is formed of: the username [0;37mcsuser[0m, the symbol [0;33m@[0m, the machine id [0;32mmetagenomics-jorge[0m, the character :, the current directory [0;36m~[0m and the character $.  Prompts are configurable (as we will see later)
 
   You enter commands after the dollar sign ($: the end of the prompt) and then press the Enter key ([1;32m↵[0m). The [0;33mCLI[0m program will then run the command you entered (if it exists). 
   For example, when you entered "csguide" to read this guide, like so:

[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ csguide[1;32m↵[0m

   you instructed the [0;33mCLI[0m to run the program "csguide" which, in turn, ran the "less" program to display the file with this content you are reading. That is, right now you are within the "less" program, moving around this guide by typing in "less" commands. Once you press the key "q", you will exit "less" and go back to the [0;33mCLI[0m which, as mentioned above, will again display its prompt for you to enter another command.

  For your information:

  • The [0;33mCLI[0m screen displays both inputs and outputs. [0;32mInputs[0m are anything you type in with the keyboard, either a command for the [0;33mCLI[0m or some data for a program (e.g., less). You can [0;32muse the mouse to copy and paste[0m some text which may be used as input, but you cannot double click (somewhere) to run a program as is typical of Windows and MacOS graphical user interfaces (GUIs) — though there are Linux GUIs like GNOME. [0;32mOutputs[0m is anything that the [0;33mCLI[0m or a program displays in the screen and includes results, prompts, and errors. The displaying of this guide is the "results" of running the "less" program. The "less" prompt is a colon [0;33m:[0m on the bottom left. If you try to run a command that does not exist, the [0;33mCLI[0m will display the error " ..: command not found".

  • The [0;33mCLI[0m is also widely known both as the [0;33mshell[0m, the [0;33mterminal[0m and the [0;33mconsole[0m. The term [0;33mCLI[0m emphasises the way you interact with Linux: by entering plain-text commands that specify the operations you want to be performed. Most commands are one-line commnands but multiple-line commands are possible too as we will see.  The term [0;33mshell[0m emphasises the role of the [0;33mCLI[0m as a communication layer, between the user and Linux services, that serves to invoke (run) those services on the user's behalf. For example, when you entered the command "..$ csguide[1;32m↵[0m", the [0;33mshell[0m requested from Linux to create a process to run (execute the code of) the "csguide" program (and likewise, "csguide" asked Linux to create a process to run the code of the "less" program) and  pass it control. The terms [0;33mterminal[0m and [0;33mconsole[0m emphasise the text-based characteristics used to display inputs and outputs, as did the terminals of the 1970s —— no fine-grain pixels can be managed throughout the screen, as in Graphical User Interfaces (GUIs), and hence no figures can be displayed. Actually, it is a character-based screen, as it is possible to display some special characters such as the bullet • and others.

  • The [0;33mCLI[0m is much more powerful than a GUI because it allows you (1) to access all the options of each program, and (2) you can make scripts, which are Bash programs that run other programs including Bash programs! GUIs usually offer only the most common options of a program and in general are not programmable. On the other hand, the [0;33mCLI[0m is somewhat more difficult to use in the beginning because you must first get acquainted with what each program does, its options and parameters, and with scripting which, basically, is computer programming. However, once you get familiar with all these, automating your tasks is not only fun but also very rewarding, and your work will be many times more efficient.    

  • You are using the [0;32mBash[0m shell whose name stands for Bourne Again SHell. There are other shells: Korn shell, C shell, etc. Bash shell is the most widely used and is the default shell in Linux. 

		    Topic 3. [37m[45mUsing CSEnv: moving around and managing files(B[m

This section shows how to run [0;33mshell[0m commands in [0;33mCSEnv[0m to help you get acquainted with the organisation of [0;33mCSEnv[0m, how to move around it, and how to create and manage files and directories. 

  This is a hands-on section: run the commands we show below as you read along. For this purpose, connect to your instance twice, so that you will have two terminals connected to [0;33mCSEnv[0m, one to continue reading this guide and another to enter the commands in the examples below. 

  Please open another terminal in your local machine (or if you are in Windows, run Git Bash again), and then login to [0;33mCSEnv[0m within the newly opened terminal (exactly as you logged in in the first terminal with ssh: "ssh -i ... csuser@..york.ac.uk").

  You can [0;32mswitch between the two terminals[0m by pressing Alt-Tab, that is: pressing simultaneously the Alt key and the Tab key. 

  This is how each example of using [0;33mshell[0m commands is presented below:
  — A few sentences of motivation for the example commands.
  — The running of the shell commands and their outputs — and sometimes comments starting with a hash #.
  — Notes on various aspects related to the commands presented to round up your knowledge of the [0;33mshell[0m.

  Example:
[1;36mExample 0. Motivation: Find out who is using your instance.[0m Please type the following command:

  ——————————————————————— [37m[44mC O M M A N D S   &   O U T P U T S(B[m —— [0;33mcomments[0m
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ who                                     # this is a comment, don't type this
csuser   pts/0        2022-02-24 22:11 (92.0.28.88)              # output of who, 2 lines
csuser   pts/1        2022-02-24 22:12 (92.0.28.88)
  ————————————————————————————————————— [37m[44mN O T E S(B[m
• The command 'who' shows who is logged on, when they logged on, and from which Internet address. 
• Your output should show two lines as above if you have opened two terminals as suggested.
• The type of terminal being used is termed 'pts', which stands for PseudoTerminal Slave device.

  [34m[47m NB (B[m
  — we are not showing the Enter [1;32m↵[0m key after each command anymore for readability, but please 
    press it after each command you are asked to type in. 

  — you only need to type what is shown after the [0;33mshell[0m prompt (after $) but not the comment (if any) 
    which starts with a hash # and continues til the end of line. 
    Thus, to follow the example above you would only type 'who' (with no quotes) and press Enter 
    in your terminal.

  [0;32mMoving around [0;33mCSEnv[0m[0m

[1;36mExample 1. Find out what is in your home directory.[0m Please type the following commands (cd, pwd, ls):

  ——————————————————————— [37m[44mC O M M A N D S   &   O U T P U T S(B[m —— [0;33mcomments[0m
                       $ cd                                      # no output of cd, but see next example
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ pwd
/home/csuser                                                     # output of pwd
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ ls
[0;33mbin  shell_data  software[0m                                        # output of ls
  ————————————————————————————————————— [37m[44mN O T E S(B[m
• The command 'cd' (Change Directory) with no arguments moves you to your home directory. 
• The command 'pwd' (Print Working Directory) prints your "working directory" (aka 
  "current directory") which is your home directory, /home/csuser, because you have just moved there.
• The command 'ls' (LiSt) with no arguments lists the files and directories in your current directory 
  (currently your home directory). 
  Your home directory has these three directories: bin, shell_data, and software.
  — [0;33mbin[0m stands for binaries, another name for programs you can run. The bin directory has a few 
    programs you may find useful —— we'll look at them later. 
  — the directory [0;33mshell_data[0m contains 'omics data to be analyed during the course.
  — the directory [0;33msoftware[0m contains software packages the Cloud-SPAN team installed.  

[1;36mExample 2. The prompt always shows your current directory.[0m Please type the following commands:

  ——————————————————————— [37m[44mC O M M A N D S   &   O U T P U T S(B[m —— [0;33mcomments[0m
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ echo This is my home directory ~
This is my home directory /home/csuser
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ cd software
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~/software[0m $ pwd
/home/csuser/software
  ————————————————————————————————————— [37m[44mN O T E S(B[m
• The command 'echo' displays what follows it, replacing variable names with their value.
• The character ~ is a [0;33mshell[0m variable that is set to the value of your home directory when you login.
  You can see in the output of 'echo' that ~ has been replaced with /home/csuser
• The command 'cd software' moves you to the software directory, changing both your current 
  directory and the prompt which now shows [0;36m~/software[0m, just before the $ symbol, as 
  the current directory.
• The command 'pwd' now shows /home/csuser/software as the current directory.
• Although 'cd' does not print any output in a new line, the prompt is changed accordingly. 

[1;36mExample 3. What is the file structure of your home directory?[0m Please type the following command:

  ——————————————————————— [37m[44mC O M M A N D S   &   O U T P U T S(B[m —— [0;33mcomments[0m
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~/software[0m $ tree -L 1
[0;33m.[0m
├── [0;33mbcftools[0m
├── [0;33mhtslib[0m
└── Miniconda3-latest-Linux-x86_64.sh

2 directories, 1 file
  ————————————————————————————————————— [37m[44mN O T E S(B[m
• The command 'tree -L 1' displays the hierarchical organisation of the file system from the 
  current directory downwards 1 level.
• '-L 1' is an option to 'tree' that specifies how many levels downwards should be printed.
  If no levels are specified, 'tree' will print all levels down. Try it: type 'tree' only.
• No argument was given to 'tree', so 'tree' assumed the current directory [0;36m~/software[0m
  as the starting point to display the hierarchy, showing two directories (in [0;33mbrown[0m) and 1 file.
• The current directory is shown at the top of the output as a single (brown) dot: [0;33m.[0m
• A dot . is an alias for (represents) the current directory. 
• Two dots .. are an alias for the parent directory of the directory.  
• Every directory has these two aliases whose purpose is to facilitate moving around the file system 
  hierarchy. You can see them if you type 'ls -a'. Try it.  
  The option '-a' to 'ls' stands for "all: do not ignore entries starting with a dot .".

  As we want to know the structure of the home directory (not the current directory ~/software), please 
type the following command which prints 2 levels downwards starting on the parent directory (..). 

  ——————————————————————— [37m[44mC O M M A N D S   &   O U T P U T S(B[m —— [0;33mcomments[0m
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~/software[0m $ tree -L 2 ..
[0;33m..[0m
├── [0;33mbin[0m 
│   ├── [1;36mbcftools[0m -> [0;32m/home/csuser/software/bcftools/bcftools[0m
│   ├── [0;32mclean[0m
│   ├── [0;32mcsguide[0m
│   ├── csguidecontent.txt
│   ├── [0;32mcsguideeditnprint.sh[0m
│   ├── [1;36mcsguide.sh[0m -> [0;32mcsguide[0m
│   ├── [0;32mfindstr[0m
│   ├── [0;32mlssr[0m
│   ├── [0;32mtextcolorsandstyles.sh[0m
│   └── [1;36mvcfutils.pl[0m -> [0;32m/home/csuser/software/bcftools/misc/vcfutils.pl[0m
├── [0;33mshell_data[0m
│   ├── [0;33msra_metadata[0m
│   └── [0;33muntrimmed_fastq[0m
└── [0;33msoftware[0m
    ├── [0;33mbcftools[0m
    ├── [0;33mhtslib[0m
    └── Miniconda3-latest-Linux-x86_64.sh

7 directories, 11 files
  ————————————————————————————————————— [37m[44mN O T E S(B[m
• In the last command ("tree -L 2 .."), the parent directory .. is an argument to the command 'tree' 
  that (1) specifies the starting point of the file structure to be displayed and (2) is displayed 
  at the top of the output.  We know the parent directory in this case is the home directory and 
  hence could have typed "tree -L 2 ~" (instead of "tree -L 2 ..") to get the same output except 
  that [0;33m/home/csuser[0m would be displayed at the top of the output. Try it please.

            [37m[45mDigression on options and arguments to commands:(B[m
            — Most options to commands start with one or two hyphens - or -- followed by a letter 
              or a word and possibly a value.
            — Most arguments to commands are names to files (data, programs, etc.) or directories. 
            — [0;31mAVOID[0m names for files and directories that start with a hyphen, for example: -hola.txt
              It may confuse the [0;33mshell[0m and some commands will not work properly.

• The output shows in the first level the 3 directories we saw before (bin, shell_data and software). 
  There are four directories in the second level: sra_metadata, untrimmed_fastq, etc.
• The files in [0;32mgreen[0m within the [0;33mbin[0m directory are programs/commands you can run.
• You are running the "csguide" which is the third program from the top.
• The files in [1;36mcyan[0m are links to other files, to the files whose name appears to the right of the 
  arrow symbol "->" in the same line. Links are like aliases and are useful in many contexts.
  We will will discuss them later in the context of linking (pointing to) programs as above. 
• The files in white contain simple text and are not runnable.

[1;36mExample 4. How to create, copy and delete files and directories.[0m At some point in your work you will want or need to make a copy of your data in order to explore "something new" without jeopardising your original data. With the commands below we are going to:
— create a new directory in your home directory 
— copy some files into the new directory
— looking at and processing the contents of copied files
— deleting files and directories 

  ——————————————————————— [37m[44mC O M M A N D S   &   O U T P U T S(B[m —— [0;33mcomments[0m 
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~/software[0m $ cd				 # go back to your home directory
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ ls				     	 # check its content again!
[0;33mbin  shell_data  software[0m
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ mkdir tmp      			 # create the tmp (temporary) directory
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ ls				     	 # check tmp has been created
[0;33mbin  shell_data  software  tmp[0m
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ cp shell_data tmp			 # copy directory shell_data to tmp
cp: -r not specified; omitting directory 'shell_data'		 # [0;31mERROR[0m 
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $ cp -r shell_data tmp			 # try again with option -r
[0;37mcsuser[0m[0;33m@[0m[0;32mmetagenomics-jorge[0m:[0;36m~[0m $					 # no output: [0;32mSUCCESS[0m
  ————————————————————————————————————— [37m[44mN O T E S(B[m
• The command 'mkdir' (MaKe DIRectory) creates a new directory. You must provide 'mkdir' the name of the directory to create as an argument. You can create two or more directories in one go, just provide the names one after the other, thus: 'mkdir t1 t2 t3'.

• The command 'cp' (CoPy) copies files and directories. The syntax is 'cp sourcefile targefile', meaning that a copy of sourcefile (1st argument) is to be made with the name targetfile (2nd argument). Thus the copy goes from LEFT to RIGHT, as the writing of English goes.

• The reason for the ERROR "omitting directory 'shell_data'" is that 'cp' was designed to copy regular files only, unless the user explicitly specifies that directories (and files within) be copied as well with the option '-r', which stands for "recursive copy". That means to repeatedly apply the copy operation on files within directories, within directories, and so on until no directories are found. 

• Our comment "no output: SUCCESS" means that, unless the purpose of a command is to print something, like the current directory ('pwd') or a list of files ('ls'), the command will not print anything unless something goes wrong. For this behaviour [0;33mshell[0m commands are said to be "silent". 

• It is good practice to use 'ls' repeateadly as shown above, so that you can check (1) that you are working in the directory you mean to be working, and (2) that things are progressing as you intend. You don't want to issue a few commands in a row, including perhaps deleting files, just to find out you did it in the wrong place. For example, suppose you want to make a new copy of file f1 onto f2, that is: 'cp f1 f2'. If (for any reason) both files exist, then 'cp' will overwrite (delete) f2 with no warning whatsoever, silently, because, by design, 'cp' works thus. If overwriting f2 was what you meant, fine. But if not, you could have avoided losing f2 by checking with 'ls' what files were in your current directory. 
  
[1;36mExample 5. Let's do the previous example more quickly using the 'Tab' key.[0m
 
		  Topic 4. [37m[45mScripting: tips and examples to help you get started(B[m

		        Topic 5. [37m[45mBackup your work: software and data(B[m

		  Topic 6. [37m[45mSome Resources: for you to go deeper into scripting(B[m

