Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

First adventure within a shell - Andrea Telatin at Quadram Institute

169 views

Published on

Slides used to introduce the Linux terminal during a short introductory course.

Published in: Engineering
  • Be the first to comment

First adventure within a shell - Andrea Telatin at Quadram Institute

  1. 1. CLIMB Workshop Andrea Telatin, Quadram Institute Biosciences Working with a Linux shell Gut Microbes and Health ISP
  2. 2. Andrea TelatinQuadram GMH Training from xkcd
  3. 3. Andrea Telatin ๏ Website • https://seq.space/notes/ ๏ Goals • Day1 — Logging into a remote server • Day2 — Understanding where are files • Day3 — Inspecting text files • Day4 — Testing first bioinformatics tools
  4. 4. Andrea Telatin ๏ Website • https://seq.space/notes/ ๏ Goals • Day1 — Logging into a remote server • Day2 — Understanding where are files • Day3 — Inspecting text files • Day4 — Testing first bioinformatics tools
  5. 5. Andrea Telatin ๏ Website • https://seq.space/notes/ ๏ Goals • Day1 — Logging into a remote server • Day2 — Understanding where are files • Day3 — Inspecting text files • Day4 — Testing first bioinformatics tools NOTE: Each tutorial is meant to be finished individually, at your pace, after the class. The first two days are tightly coupled: don’t worry if you can’t finish today’s tutorial
  6. 6. what is a shell?
  7. 7. Andrea TelatinQuadram GMH Training The UNIX Terminal • it’s a powerful text interface to interact with PCs and servers • carries a long history of evolution and improvements from the ‘70 • it’s everywhere: most websites are hosted in a Linux server. Any Mac runs on UNIX. 
 It matters to us: bioinformatics loves Linux.
  8. 8. Andrea TelatinQuadram GMH Training The shell • It’s a program that allows us to interact with the system: list files, run other programs… • DOS was a shell by Microsoft. We are interested in UNIX compatible shells, there are many the most commonly found in servers being Bash • Bash can replace whatever you are used to do under Windows with a mouse, and more.
  9. 9. Andrea TelatinQuadram GMH Training The shell
  10. 10. Andrea TelatinQuadram GMH Training The shellPrompt
  11. 11. Andrea TelatinQuadram GMH Training The shellPrompt
  12. 12. Andrea TelatinQuadram GMH Training The shell Your command Prompt
  13. 13. Andrea TelatinQuadram GMH Training The shell Your command Prompt
  14. 14. Andrea TelatinQuadram GMH Training The shell Your command Prompt Output from the program
  15. 15. Andrea TelatinQuadram GMH Training The shell
  16. 16. Andrea TelatinQuadram GMH Training The shell ‣ How many commands are present? ‣ Did some program return no output?
  17. 17. remote servers
  18. 18. Laptop by Edward Boatman from the Noun Project andrea’s client john’s client GMH
 Server
  19. 19. Laptop by Edward Boatman from the Noun Project andrea’s client john’s client GMH
 Server Benefits: servers are… • shared resources: multiple users can access the same data (reducing the need and risks of duplications) • powerful resources: servers carry more computational power for the same cost of a laptop • reliable resources: redundant power supply, redundant disks. Higher availability and data is more secure • managed resources: we don’t need to remember to backup our data, someone else will.
  20. 20. Andrea TelatinQuadram GMH Training Connecting to a remote server ➡ From Linux or OS X • Open your terminal • Type ssh remoteuser@remoteaddress ➡ From Windows • You need a free terminal emulator called PuTTY (4ngs.com/go/hG)
  21. 21. Andrea TelatinQuadram GMH Training Connecting to a remote server ➡ From Linux or OS X • Open your terminal • Type ssh remoteuser@remoteaddress ➡ From Windows • You need a free terminal emulator called PuTTY (4ngs.com/go/hG)
  22. 22. Andrea TelatinQuadram GMH Training Connecting to a remote server ➡ From Linux or OS X • Open your terminal • Type ssh remoteuser@remoteaddress ➡ From Windows • You need a free terminal emulator called PuTTY (4ngs.com/go/hG)
  23. 23. Andrea TelatinQuadram GMH Training Connecting to a remote server ➡ From Linux or OS X • Open your terminal • Type ssh remoteuser@remoteaddress ➡ From Windows • You need a free terminal emulator called PuTTY (4ngs.com/go/hG)
  24. 24. putty.exe
  25. 25. 1 2 ubuntu@137.44.55.66 Quadram Workshop 3 4 putty.exe
  26. 26. Andrea TelatinQuadram GMH Training Connecting to a remote server • When typing password inside a terminal you won’t see anything! Just type then hit enter! • The first time you access to a remote server you’ll be asked if you trust it (message in the terminal or windows from PuTTY)
  27. 27. Andrea TelatinQuadram GMH Training Connecting to a remote server • Once logged into the remote server you’ll only be able to see the files present there • It’s possible to copy files from your local machine (client) to the remote machine (server) • It’s a good idea to always use some server for our work: clients are never as safe and reliable as a server.
  28. 28. “screen” safety net
  29. 29. andrea’s client GMH
 Server$ command SSH Session: a constant flow of text going to and coming from the remote server
  30. 30. andrea’s client GMH
 Server Even a small interruption in the connectivity will make the server assume we are no longer interested!
  31. 31. andrea’s client GMH
 Server We can use a program called screen to keep the session alive even if we close our computer!
  32. 32. andrea’s client GMH
 Server We can use a program called screen to keep the session alive even if we close our computer!
  33. 33. Andrea TelatinQuadram GMH Training The “screen cycle” • Check if there are screens already
 screen -list • Resume the existing screen (if present)
 screen -dr • Create a new screen otherwise
 screen -S SessionName
  34. 34. File Paths a.k.a. the most difficult topic of this course!
  35. 35. Andrea TelatinQuadram GMH Training Concepts • Current directory, or working directory, is the place you are currently operating from. You always are operating from a specific directory. • Each file has a unique position in the filesystem tree (like a leaf in a taxonomy tree). You can describe it in two ways: absolute or relative.
  36. 36. Andrea TelatinQuadram GMH Training Concepts • Absolute paths, like absolute coordinates, are the same for any user of the system, no matter the current working directory. They always start with / • Relative paths are generated assuming your working directory as the starting point. They never start with /.
  37. 37. / home/ etc/ tmp/ usr/ bin/ Proch/ Lizzy/ my_file.txt
 linux.pdf perl report.txt
 linux.pdf Working Dir
  38. 38. / home/ etc/ tmp/ usr/ bin/ Proch/ Lizzy/ my_file.txt
 linux.pdf perl report.txt
 linux.pdf Working Dir cd Proch
  39. 39. / home/ etc/ tmp/ usr/ bin/ Proch/ Lizzy/ my_file.txt
 linux.pdf perl report.txt
 linux.pdf Working Dir cd Proch cd /home/Proch
  40. 40. / home/ etc/ tmp/ usr/ bin/ Proch/ Lizzy/ my_file.txt
 linux.pdf perl report.txt
 linux.pdf Working Dir
  41. 41. / home/ etc/ tmp/ usr/ bin/ Proch/ Lizzy/ my_file.txt
 linux.pdf perl report.txt
 linux.pdf Working Dir Proch/
  42. 42. / home/ etc/ tmp/ usr/ bin/ Proch/ Lizzy/ my_file.txt
 linux.pdf perl report.txt
 linux.pdf Working Dir Proch/
  43. 43. / home/ etc/ tmp/ usr/ bin/ Proch/ Lizzy/ my_file.txt
 linux.pdf perl report.txt
 linux.pdf Working Dir Proch/
  44. 44. / home/ etc/ tmp/ usr/ bin/ Proch/ Lizzy/ my_file.txt
 linux.pdf perl report.txt
 linux.pdf Working Dir Proch/
  45. 45. / home/ etc/ tmp/ usr/ bin/ Proch/ Lizzy/ my_file.txt
 linux.pdf perl report.txt
 linux.pdf Working Dir ls -l ../Lizzy/linux.pdf Proch/
  46. 46. / home/ etc/ tmp/ usr/ bin/ Proch/ Lizzy/ my_file.txt
 linux.pdf perl report.txt
 linux.pdf Working Dir . Current directory .. Upper directory
 ~ Home directory ls -l ../Lizzy/linux.pdf Proch/
  47. 47. Andrea TelatinQuadram GMH Training Commands history • Using the up arrow you will recall the last command typed, hit it again and you get the previous and so on. • Down arrow is working as well • The command history will
 print the list of commands
 issued so far
  48. 48. Andrea TelatinQuadram GMH Training Commands history • Using the up arrow you will recall the last command typed, hit it again and you get the previous and so on. • Down arrow is working as well • The command history will
 print the list of commands
 issued so far
  49. 49. Andrea TelatinQuadram GMH Training Tab completion • When you are typing a file path (relative or absolute) you are likely to introduce errors. • Most shells have a key (Tab) to help you type a correct path. • It’s thus mandatory to use the autocompletion
  50. 50. Andrea TelatinQuadram GMH Training Tab completion • When you are typing a file path (relative or absolute) you are likely to introduce errors. • Most shells have a key (Tab) to help you type a correct path. • It’s thus mandatory to use the autocompletion
  51. 51. Panic management • Breaks current work (exit) • After killing the current process, you’ll get the prompt back • Also remember that Ctrl + D breaks a stream of data
  52. 52. Panic management • Breaks current work (exit) • After killing the current process, you’ll get the prompt back • Also remember that Ctrl + D breaks a stream of data
  53. 53. Anatomy of a command
  54. 54. Andrea TelatinQuadram GMH Training Structure of a command • command parameters … • ls -l /workshop • grep -c ">" file-name.txt
  55. 55. Andrea TelatinQuadram GMH Training Structure of a command • command parameters … • ls -l /workshop • grep -c ">" file-name.txt
  56. 56. Andrea TelatinQuadram GMH Training Structure of a command • command parameters … • ls -l /workshop • grep -c ">" file-name.txt
  57. 57. Andrea TelatinQuadram GMH Training Structure of a command • command parameters … • ls -l /workshop • grep -c ">" file-name.txt
  58. 58. Andrea TelatinQuadram GMH Training Structure of a command • command parameters … • ls -l /workshop • grep -c ">" file-name.txt
  59. 59. Andrea TelatinQuadram GMH Training Structure of a command • command parameters … • ls -l /workshop • grep -c ">" file-name.txt
  60. 60. Andrea TelatinQuadram GMH Training Structure of a command • command parameters … • ls -l /workshop • grep -c ">" file-name.txt Example: 4ngs.com/go/hH
  61. 61. Andrea TelatinQuadram GMH Training Structure of a command • command parameters … • ls -l /workshop • grep -c ">" file-name.txt Example: 4ngs.com/go/hH
  62. 62. Redirection and pipes
  63. 63. Andrea TelatinQuadram GMH Training Redirection • Programs often print some text on our terminal. If we want we can redirect that text from the terminal to a file (i.e. save to a file the output) • ls -l > my_list.txt
 find / > all_files.txt 2> errors.log
  64. 64. Andrea TelatinQuadram GMH Training Piping commands • A method to create a flow of data: the output of the upstream command becomes the input of the downstream program. We will see this later. • find ~ | wc -l
  65. 65. Some commands
  66. 66. Andrea TelatinQuadram GMH Training Where am I: pwd • Means “print working directory”, no parameters to be supplied. • It returns the absolute path of our current position • It’s important to check if we are where we think we are (see also ls)
  67. 67. Andrea TelatinQuadram GMH Training Change directory: cd • cd (Change directory) allows to set a new current directory • Syntax: cd new_path • Path can be absolute:
 /home/telatin/Desktop/directory/ • or relative:
 ../../destination
  68. 68. Andrea TelatinQuadram GMH Training Change directory: cd • Without parameters goes to the home directory
 E.g.: cd • There is a shortcut for the home directory: ~ 
 E.g.: cd ~/Desktop/ • Returning to the previous working directory:
 cd -
  69. 69. Andrea TelatinQuadram GMH Training List files: ls • “ls” (list) it’s a very important command, as it replace the common file windows. • ls alone lists the file in my working directory • As argument I can give one or more directory, and “ls” will print their content (e.g.: ls /tmp) • Also files can be arguments, also with wildcards
 e.g.: ls *.fa *.fastq

  70. 70. Andrea TelatinQuadram GMH Training • Switches can be combined (-l -a = -la) • -l detailed list • -a includes hidden files • -h file size given in human readable format • man ls detailed manual • How can I sort by date? OR by size? List files: ls
  71. 71. Andrea TelatinQuadram GMH Training Wildcards • ? substitutes a single character • * substitutes any string of characters • Examples: 
 ls *.fastq
 ls *_R?_*.fastq
  72. 72. Andrea TelatinQuadram GMH Training Characters range • [a-z] single lowercase letter • [A-Z] single uppercase letter • [0-9] single digit • E.g.: ls *_R[1-2]_*.fastq
  73. 73. Andrea TelatinQuadram GMH Training Copy a file cp • Syntax: cp [origin] [destination] • Origin can be one or more files. • -r to recursively copy directories and their content • man cp as always…
  74. 74. Andrea TelatinQuadram GMH Training Create directories mkdir • Syntax: mkdir [newDirName] … • -p to create intermediate directories if necessary:
 mkdir -p day1/data/sample • man mkdir for further details
  75. 75. Andrea TelatinQuadram GMH Training Remove directories rmdir • Syntax: rmdir [DirName] … • Only removes empty directories (safe) • man rmdir
  76. 76. Andrea TelatinQuadram GMH Training • Syntax: rm [file] … • Removes files, not directories, by default • -r also removes directories and their content (😵) • Careful with this one, it hurts! Delete files rm
  77. 77. How to do it the right way Before running a dangerous command
 eg: rm ./files/*.gz TEST it with an “ls” e.g.: ls ./files/*.gz
  78. 78. Andrea TelatinQuadram GMH Training • Syntax: cat [file1] [file2]… • Will print all the content of one or multiple text files • If the file is very long… can be a problem! View text files cat
  79. 79. Andrea TelatinQuadram GMH Training • Syntax: head [file1] [file2]… • Will print the first ten lines of a file • head -n 16 [file1] [file2]… • Similarily, tail will print the last lines head and tail
  80. 80. Andrea TelatinQuadram GMH Training • Syntax: man command • Manual for shell commands • You can scroll with arrow keys and quit with q • g and G to quickly reach the top/bottom • / to start searching • n N to go to the next / previous occurrence man
  81. 81. Andrea TelatinQuadram GMH Training • Syntax: less [file1] • An interactive viewer of text files. You can scroll with arrow keys and quit with q. Works like man! • To prevent word wrap: • less -S [file1] less
  82. 82. Andrea TelatinQuadram GMH Training wc: word count lines, words, characters
  83. 83. Andrea TelatinQuadram GMH Training wc: word count lines, words, characters

×