System integration in free software
Upcoming SlideShare
Loading in...5
×
 

System integration in free software

on

  • 3,756 views

 

Statistics

Views

Total Views
3,756
Views on SlideShare
3,755
Embed Views
1

Actions

Likes
0
Downloads
49
Comments
0

1 Embed 1

http://www.slideshare.net 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

System integration in free software System integration in free software Presentation Transcript

  • Systems Integration on Free Software Enrique Ocaña González Master on Free Software
  • Index (1/4) ● Administrative tasks – System startup – Users & groups – Task automating – Auditing – Backups Master on Free Software
  • Index (2/4) ● Basic commands – Files & directories – Process control – Documentation – Redirections & pipes Master on Free Software View slide
  • Index (3/4) ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software View slide
  • Index (4/4) ● Advanced commands – Regular expressions – Grep – Sed – Awk Master on Free Software
  • Administrative tasks Master on Free Software
  • Index ● System startup ● Users & groups ● Task automating ● Auditing ● Backups Master on Free Software
  • Introduction ● Administration: security – Two roles ● User: Not concerned about administrative tasks ● Administrator (root): Responsible for the proper operation of the system – System designed thinking in program and user data separation – Advice: Only work as root when needed ● Avoids unintended damage of the system Master on Free Software
  • Introduction ● Administration: transparency – Config and utilities based on plain text files and scripts – Looking at the configuration, reading documentation and following what is done by the scripts, it's possible to understand what the system does – The root of a problem can be found in a simpler way than in closed systems, where the internal system operation isn't known Master on Free Software
  • Introduction ● Administration: support – User community at internet – Specialized companies Master on Free Software
  • Introduction ● Steps to solve a problem – Read documentation ● /usr/share/doc/package ● Project website ● FAQs and howtos – Upgrade the program to the latest version (maybe the bugs have been solved) – Get in touch with the developers – Return the favour colaborating in projects ;-) Master on Free Software
  • Index ● System startup ● Users & groups ● Task automating ● Auditing ● Backups Master on Free Software
  • System startup ● System startup sequence: 1.BIOS load, POST tasks, boot sequence selection 2.Boot sector load (MBR) 3.Boot loader: grub, lilo, ntldr 4.Kernel load: /boot/vmlinuz 5. init program execution 6.System tasks: /etc/init.d/rcS 7.Runlevel specific tasks: /etc/rcN.d Master on Free Software
  • System shutdown ● System shutdown – Init process receives a state change signal that lets stopping (runlevel 0) or restarting (runlevel 6) the system ● shutdown ● halt ● init – Executes the actions associated to the new runlevel in a stablished order Master on Free Software
  • System shutdown ● System shutdown stages: 1. Shutdown warning to users and services 2. Execution of actions associated to the system shutdown (runlevel change) 3. Proper filesystem unmounting 4. Computer power off or reset Master on Free Software
  • Init and runlevels ● Init process: – Father or all the processes (PID 1) – Has different states or runlevels with services and actions associated, to be executed when entering into the runlevel or leaving it – The most common runlevels are: ● 0  System shutdown ● 6  System restart ● 1  Single user mode ● 2-5  Multiuser (2 by default) Master on Free Software
  • Init and runlevels ● Configuration files: – Master init configuration: /etc/event.d ● Used to be at: /etc/inittab – Real init scripts placed at /etc/init.d – System bootstrap init links: /etc/rcS.d – Runlevel links: /etc/rcN.d (N=0...6) Master on Free Software
  • Init and runlevels ● Symbolic links format: – S|K + order + service name ● Eg. at rc2.d: S89cron ­> ../init.d/cron ● S  Start the service when entering the runlevel ● K  Kill the service when entering the runlevel ● Manual startup: – /etc/init.d/service {start|stop| restart|reload|force­reload} – /etc/init.d/service shows help Master on Free Software
  • Boot manager ● Common boot managers: – lilo – grub i Additional documentation: boot_loader_config.pdf Master on Free Software
  • Index ● System startup ● Users & groups ● Task automating ● Auditing ● Backups Master on Free Software
  • Users & groups ● root: the user with the highest privileges ● Groups allow granting and grouping privileges of a set of users ● UID, GID: Unique numerical identifiers – root has UID 0 ● Base directory or “home” – /home/user – /root Master on Free Software
  • Users & groups ● Files involved: – /etc/passwd: Information about the user accounts ● Login ● Password – Crypted, “x” when using shadow passwords, “!” when the account is disabled ● UID ● Home directory path ● Shell ● Name, surname, other identification info Master on Free Software
  • Users & groups ● Files involved: – /etc/shadow: Like passwd, but only readable by root and meant to store the passwords – /etc/group: Info about groups and which users belong to each one – /etc/gshadow: Non public info about the groups Master on Free Software
  • User management ● User suplantation su user ● User task suplantation (“run as”) sudo user command ● Suplantation access control – File /etc/sudoers (don't touch!) – visudo Master on Free Software
  • User management ● User creation – adduser user ● Creation process 1. Entry in passwd file 2. Entry in shadow file 3. Creation of /home/user 4. Copy of the files in /etc/skel 5. Password setting Master on Free Software
  • User management ● User deletion – deluser user ● Creation process 1. Delete entries in passwd and shadow files 2. Finds all the files outside the home directory and deletes them, making a security copy Master on Free Software
  • Group management ● Adding a group – addgroup group ● Deleting a group – delgroup group ● Process: Adds/deletes the entries in the files /etc/group and /etc/gshadow Master on Free Software
  • Group management ● Adding a user to an existing group – adduser user group ● Deleting a group – adduser user group ● Process: Adds/deletes the entries in the files /etc/group and /etc/gshadow Master on Free Software
  • Password changing ● Change the password of a user – passwd  Changes the password of the current user – passwd user group  Executed as root, changes the password of the user specified ● Executed by a normal user, the command asks for the old password ● Executed by root, it doesn't Master on Free Software
  • User monitoring ● Each time a user logs into the system, at some logs are recorded: – /var/log/syslog  Generic log for all the system events. Maybe also logged on auth. – /var/log/wtmp  Specific log for all the user logins in the system. ● Binary file – /var/log/btmp  Specific log for all the failed user logins in the system. Master on Free Software
  • User monitoring ● Commands – who (or w)  Shows who is currently logged into the system and what are they doing – last / lastb  All the users that logged into the system recently. Lastb shows info of failed attempts – slay  Kicks a user out of the system and kills all her processes Master on Free Software
  • Index ● System startup ● Users & groups ● Task automating ● Auditing ● Backups Master on Free Software
  • Task automating ● cron  System utility that allows launching periodic processes (tasks, actions...) in an unattended way, at a scheduled date ● at  System utility that allows to program the execution of a process, but without repetition. It allows to defer the unattended process until a date Master on Free Software
  • Repeated tasks: cron ● The cron service consists of 2 programs: – crond  Program (daemon) offering the service to the system – crontab  Program that allows an user to manage her tasks ● crontab ­e  Edit the scheduled tasks of the current user ● crontab ­l  List the tasks of the current user ● crontab ­r  Remove all the tasks of the current user Master on Free Software
  • Repeated tasks: cron ● Crontab edition: – crontab ­e  Opens an editor predefined by the variable EDITOR, to edit the user crontab ● Tip: export EDITOR=nano – Each crontab line has the following format, always split by spaces ● Minute: Integer number between 0 and 59 ● Hour: Integer in the range 0-23 ● Day: Integer in the range 1-31 (must be a valid day if the month is also specified) Master on Free Software
  • Repeated tasks: cron ● Crontab edition: ● Month: Integer in the range 1-12 ● Week_day: Integer in the range 0-7, where 0 and 7 mean Sunday. The short name of the week day can also be used (mon, tue, wed...) ● Command: Command to be executed. The command can be something like “ls /proc >>  /tmp/proc” or a command that executes a customized script written by the user Master on Free Software
  • Repeated tasks: cron ● Moreover... – A “*” in any field indicates all valid values for that field (respecting the restricions of the rest of the fields) – A “–“ between numbers means an interval between those numbers – A list of values split by “,” means a value list – A “/” means step values. Eg: “*/15” would mean that the task is executed each 15 minutes. Master on Free Software
  • Repeated tasks: cron ● Moreover... – Lines starting with “#” are comments and aren't had into account – A task is executed in the moment when all the requirements of the schedule parameters are met Master on Free Software
  • Repeated tasks: cron ● Moreover... – Special parameters (instead of the 5 said fields) ● @reboot  One time when the computer boots ● @yearly  Once a year, eq. to 0 0 1 1 * ● @daily  Each day, eq. to 0 0 * * * ● @weekly  Each week, eq. to 0 0 * * 0 ● @monthly  Each month, eq. to 0 0 1 * * ● @hourly  Each hour, eq. to 0 * * * * Master on Free Software
  • Repeated tasks: cron ● Examples (crontab -e)   # min hour day month weekday command   # Run 5 min after midnight, every day 5 0 * * * /bin/daily.sh >> /tmp/out 2>&1   # Run at 2:15pm on the first of every month 15 14 1 * * $HOME/bin/monthly   # Run at 10pm on weekdays 0 22 * * 1­5 /usr/local/bin/script.sh   # Run 23 min after midnight, 2am, 4am... everyday 23 0­23/2 * * * script.sh Master on Free Software
  • Delayed tasks: at ● at allows to defer the execution of a task. Usage:  at date task CTRL+d ● Example:  $ at 4:45pm at> echo “Time to go” CTRL+d Master on Free Software
  • Delayed tasks: at ● atq  Shows pending tasks  # atq 1  2008­01­01 10:35 b peter 2  2008­02­15 00:00 c peter 5  2008­02­10 01:00 Z root ● at -c 2  Shows all the execution environment for task 2 ● atrm tasknum  Deletes the specified task from the task queue Master on Free Software
  • Delayed tasks: at ● Considerations – Admin can control which users cat use at: /etc/at.allow, at.deny. She can also see the tasks of all users – at has a queue system to set the priority of the tasks – batch is a command derived form at that allows executing a task only when the system load goes below of 0.8 Master on Free Software
  • Index ● System startup ● Users & groups ● Task automating ● Auditing ● Backups Master on Free Software
  • Log location and format ● At log files are located at /var/log ● Important files – messages  System wide messages – syslog  Messages gathered by the log daemon (syslogd) – debug  System logs useful to solve problems – auth.log  Login and logoff attempts Master on Free Software
  • Log location and format ● Log format Oct 26 18:17:01 localhost CRON[3384]: (pam_unix) session opened... Log line Process name and PID Computer Date and time ● Log continuous viewing tail ­f logfile Master on Free Software
  • Log location and format ● Log files with a non-standard format – dmesg  Last system startup info – faillog  Info about the last time users attempted to log into the system (fail number, max allowed, last fail). ● Viewed using faillog ­u user – lastlog  List of users and the last time they logged into the system. ● Viewed using lastlog Master on Free Software
  • Log location and format ● Log files with a non-standard format – Xfree86.0.log, Xorg.0.log  Info from the X-Window system. ● Viewed with standard viewers (tail, etc.) i http://www.debian.org/doc/manuals/users-guide/ch-rsi.en.html Master on Free Software
  • Log analysis and tools ● Log analysis allows to detect intrusions in the computer and hardware/software problems ● logcheck  Looks for strange log entries and sends them to the administrator via email i logcheck.pdf Master on Free Software
  • Log configuration ● Log file configuration: /etc/syslog.conf ● Format: – Selector: service.priority [[, service...]; service.priority...] ● Services: auth, authpriv, cron, daemon, ftp, kern, lpr, news, syslog, user, uucp, local0...7, *, none ● Priorities: debug, info, notice, warning, crit, alert, emerg, *, none – =priority: only that priority, not higher ones – !priority: only that priority, not lower ones – !=priority: exactly that priority Master on Free Software
  • Log configuration ● Format: – Action ● /.../file: Log to log file (use complete path) ● |pipe: Log to named pipe ● /dev/console: Log to the console ● @hostname: Log to a remote computer ● user1, user2, ...: Send to a list of users ● *: Send to all the users Master on Free Software
  • Log configuration ● Example: kern.*                /var/adm/kernel kern.crit             @earth kern.crit             /dev/console kern.info;kern.!err   /var/adm/kernel­info mail.*;mail.!=info    /var/adm/mail *.*                   @mars i man syslog.conf, man syslogd Master on Free Software
  • Log configuration ● Log rotation: logs are rotated periodically. They are compressed and deleted by age order ● Example: – /var/log/syslog – /var/log/syslog.0 – /var/log/syslog.1.gz – /var/log/syslog.2.gz Master on Free Software
  • Log configuration ● Configuración ● /etc/logrotate.conf: Main log rotation configuration ● /etc/logrotate.d/*: Specific configuration for particular services i man logrotate Master on Free Software
  • Index ● System startup ● Users & groups ● Task automating ● Auditing ● Backups Master on Free Software
  • Backup policy ● Backup levels – Level 0: Complete backup – Level 1: Incremental, changes from level 0 – Levels 2..9: Changes from previous level ● Storage media – Tape – CD, DVD – Hard disk – Network Master on Free Software
  • Backup policy ● Backup calendar – Diary complete Backup level M T W T F S S 0 0 0 0 0 0 0 – Weekly complete, diary level 1 M T W T F S S 1 1 1 1 1 1 0 – Weekly complete, diary level 1 M T W T F S S 1 2 3 4 5 6 0 Master on Free Software
  • Backup policy ● Backup calendar – Monthly hanoi tower M T W T F S S 3 2 5 4 7 6 1 3 2 5 4 7 6 1 3 2 5 4 7 6 1 3 2 5 4 7 6 0 i http://www.backupschedule.net/backupSchedules/towerOfHanoi.html Master on Free Software
  • Automating and scripts ● Ad-hoc backup: tar ● Saving tar zcvf file.tgz files... gzip create verbose file ● Restoring tar zxvf file.tgz [files...] extract Master on Free Software
  • Automating and scripts ● Ad-hoc backup: tar – Other functions ● ­A: Concatenate some tar files (not tgz!) ● ­d: Find differences between tar/tgz file and files on disk ● ­­delete: Delete files in tar/tgz file ● ­r: Append files at the end of a tar/tgz file ● ­t: List contents ● ­u: Only append files newer than the ones in the tar/tgz file Master on Free Software
  • Automating and scripts ● Other useful commands – cpio: Input/output copy to file find . ­print ­depth | cpio ­ov > tree.cpio cpio ­idv < tree.cpio – rsync: Remote directory synchronization rsync ­aP ­e ssh user@host:rem_path loc_path – dd: Exact copy from one device to another (or to a file) dd if=/dev/hda1 of=bak_hda1.bin Master on Free Software
  • Automating and scripts ● Other useful commands – mt: Magnetic tape management mt ­f /dev/st0 eject mt retension mt status – Example of an ad-hoc backup on tape #!/bin/sh # SYSTEM CONFIG AND DATA BACKUP tar jcvlf ­ ­­exclude=/var/www /var /etc   | buffer ­B ­s 10k > /dev/st0 mt offline Master on Free Software
  • Automating and scripts ● Simple automating: cron ● Advanced backup utilities: Amanda – Backup automating – Network backup server, multiple client platforms (even Windows using Samba) – Mail reminder for administrators – Tape dump – Intelligent estimation of backup leven based on previous statistics Master on Free Software
  • Automating and scripts ● Amanda configuration – Multiple configs. Eg: /etc/amanda/DailySet1 – Authorized clients: /etc/amandahosts – Main config: amanda.conf – List of disks to be dumped: disklist – List of tapes to be used: tapelist (don't touch!) Master on Free Software
  • Automating and scripts ● Tape labeling for Amanda use amlabel [­f] DailySet1 DMP00 Force, for a Configuration to Tape identifier (syntax defined previously be used at amanda.conf) labeled tape Master on Free Software
  • Automating and scripts ● Recovering: amrecover root@cesar:~ # cd / root@cesar:/ # amrecover ­C DailySet1 ­s localhost.localdomain AMRECOVER Version 2.4.4p3. Contacting server on localhost ... The files are going to be extracted in the 220 localhost.localdomain AMANDA index server (2.4.4p3) ready. 200 Access OK current directory. It's important to change to Setting restore date to today (2004­11­03) computer to be the root directory (“/”) of the restored, in particular if it's a satellite 200 Working date set to 2004­11­03. ... computer (not the one hosting the backup) amrecover> listdisk 200­ List of disk for host localhost.localdomain 201­ /home 201­ /etc 201­ /var 200 List of disk for host localhost.localdomain amrecover> setdisk /var 200 Disk set to /var. Master on Free Software
  • Automating and scripts ● Recovering: amrecover root@cesar:~ # cd / root@cesar:/ # amrecover ­C DailySet1 ­s localhost.localdomain AMRECOVER Version 2.4.4p3. Contacting server on localhost ... 220 localhost.localdomain AMANDA index server (2.4.4p3) ready. 200 Access OK Setting restore date to today (2004­11­03) NOTE: check access 200 Working date set to 2004­11­03. permissions at ... /etc/amandahosts amrecover> listdisk 200­ List of disk for host localhost.localdomain 201­ /home 201­ /etc 201­ /var 200 List of disk for host localhost.localdomain amrecover> setdisk /var 200 Disk set to /var. Master on Free Software
  • Automating and scripts ● Recovering: amrecover amrecover> setdate 2004­10­20 200 Working date set to 2004­10­20. amrecover> ls ... 2004­10­13 spool/ 2004­10­13 www/ amrecover> add www Added dir /www at date 2004­10­13 amrecover> list TAPE DailySet5      LEVEL 0 DATE 2004­09­30   /www TAPE /home/amand... LEVEL 1 DATE 2004­10­13   /www amrecover> extract ... i man amanda , /usr/share/doc/amanda , http://www.amanda.org Master on Free Software
  • Basic commands Master on Free Software
  • Index ● Basic commands – Files & directories – Process control – Documentation – Redirections & pipes Master on Free Software
  • Index ● Basic commands – Files & directories – Process control – Documentation – Redirections & pipes Master on Free Software
  • Files basic commands ● I-node: Data structure in a filesystem. Stores basic data about the file, directory or another object in the filesystem – Size and physical location (disk block) – Owner and group – Mode (permissions) – Creation, modification and last access time – Reference counter, accounting how many hard links are pointing to the inode Master on Free Software
  • Files basic commands ● Example: $ ls ­lisa total 16 766122 4 drwxr­xr­x   2 root  root  4096 . 765537 4 drwxrwxrwx  11 root  root  4096 .. 766125 4 ­rw­r­­r­­   2 root  root     5 file1 766125 4 ­rw­r­­r­­   2 root  root     5 file2 766126 0 lrwxrwxrwx   1 root  root     8 file3 ­> file1 Inode Mode Owner and number group Used blocks Reference Used bytes counter Master on Free Software
  • Files basic commands ● Permissions (mode) – Permissions ● Read (r): 4 (octal) 750 ● Write (w): 2 Others: (no access) ● Execution (x): 1 – Granularity Group: 4(read)+1(execution) ● User (u) User: 4(read)+2(write)+1(execution) ● Group (g) ● Others (o) ● All (a): ugo Master on Free Software
  • Files basic commands ● Special permissions (4 nibble) th ! UNSAFE!! – Setuid bit (o+s, 4): Sets the owner of the file as the effective user when the file is executed – Setgid bit (g+s, 2): Sets the group of the file as the effective user when executed – Sticky bit (STIcky, Save Text Image) (o+t, 1) ● For files: Keeps the code in swap, just if it's executed later ● For directories: Only the owner of the files inside and the owner of the directory can delete files in the directory Master on Free Software
  • Files basic commands ● Links – Hard links: The file has two entry points at the same time – Soft links: Reference to the true path of the file a b c d... a b c d... 1234... 1234... Master on Free Software
  • Files basic commands ● Example $ ls ­lisa total 16 766122 4 drwxr­xr­x   2 root  root  4096 . 765537 4 drwxrwxrwx  11 root  root  4096 .. 766125 4 ­rw­r­­r­­   2 root  root     5 file1 766125 4 ­rw­r­­r­­   2 root  root     5 file2 766126 0 lrwxrwxrwx   1 root  root     8 file3 ­> file1 a b c d... a b c d... 1234... 1234... Master on Free Software
  • Files basic commands ● Example $ ls ­lisa total 16 766122 4 drwxr­xr­x   2 root  root  4096 . 765537 4 drwxrwxrwx  11 root  root  4096 .. 766125 4 ­rw­r­­r­­   2 root  root     5 file1 766125 4 ­rw­r­­r­­   2 root  root     5 file2 766126 0 lrwxrwxrwx   1 root  root     5 file3 ­> file1 Reference counter. file1 and file2 are actually two hard links to the same content, thus its reference File the symbolic link is counter is 2 pointing to Byte size (note that “file1” string takes actually 5 bytes) Master on Free Software
  • Files basic commands ● Example $ ls ­lisa total 16 766122 4 drwxr­xr­x   2 root  root  4096 . 765537 4 drwxrwxrwx  11 root  root  4096 .. 766125 4 ­rw­r­­r­­   2 root  root     5 file1 766125 4 ­rw­r­­r­­   2 root  root     5 file2 766126 0 lrwxrwxrwx   1 root  root     5 file3 ­> file1 The current directory (.) has 2 references: Entry (.) of that directory and entry mydirname in its parent directory. Parent directory (..) has 11 references: the entry tmp in the root directory and the “..” entries in the other 10 directories it has. Master on Free Software
  • Files basic commands ● Special files – Devices (high number, low number) ● Block brw­rw­­­­ 3, 0 /dev/hda ● Character crw­rw­­­­ 9,96 /dev/st0a – Named pipes (FIFO, socketpairs) srwxrwxrwx /tmp/.X11­unix/X0 Master on Free Software
  • Files basic commands ● Special directories: “.”, “..” ● Hidden files: Start by “.”. Eg: .bashrc ● Brief of the file types: d  Directory p  Pipes l  Link s  Sockets c  Char. device ­  Ordinary b  Block device Master on Free Software
  • Files basic commands ● pwd  Shows the current absolute path ● cp  Copy files cp [­ir] ( file file | dir dir | filelist  dir ) ● cd  Change directory cd path ● mv  Move or rename files and directories mv file newfile Master on Free Software
  • Files basic commands ● mkdir  Create directories mkdir [­p] directory ● ls  Shows the content of a directory or info about a file ls [­laR] [dirname]|[filename] ● rmdir  Deletes empty directories rmdir directory Master on Free Software
  • Files basic commands ● rm  Deletes files rm [­ir] file1 file2... ● find  Searchs files by name, etc. find [searchdir] ­name “filename” ● cat  Shows the content of a file without being able to edit it and without pauses cat file Master on Free Software
  • Files basic commands ● less/more  Shows the content of a file without being able to edit it but page by page less file ● chmod  Permission (mode) change chmod [options] mode file – Examples: ● chmod u+srwx,g­w,o+rxt prueba ● chmod 5755 prueba Master on Free Software
  • Files basic commands ● ln  Link creation – Hard link: Not allowed among directories ln sourcefile destinationlink – Soft link ln ­s sourcefile destinationlink Master on Free Software
  • Index ● Basic commands – Files & directories – Process control – Documentation – Redirections & pipes Master on Free Software
  • Processes ● Linux is multitasked – Each instance of a command or program being executed is a process ● The kernel controls all the processes – Allocates resources (memory, cpu,...) – Schedules per priority the processes to be executed – Controls resource access. Each process has an owner – Process info at /proc Master on Free Software
  • Processes ● Important properties characterizing a process – Process identification number (PID) – Parent id number (PPID) – State: Running, Sleep (waiting for a resource) – Owner: Usually the user running the process – Priority: importance in respect to other processes Master on Free Software
  • Process control commands ● ps  Lists running processes (snapshot) ps ­aux Shows user, PID, %CPU, %MEM, STAT, time... ● top  Interactively lists running processes. Useful for monitoring. Allows process management ● kill  Sends a termination signal kill ­KILL pid Causes the abrupt finishing of the process with PID pid Master on Free Software
  • Process control commands ● nice  Starts process with given priority. Range from -20 (higher) to 19 (lower) nice +10 process ● snice, renice ● bg (shell)  Sends process to background – Same as & when launching from shell ● fg (shell)  Brings a process to foreground. Opposite to bg Master on Free Software
  • Process control commands ● jobs (shell)  Show processes in background for the current user ● CTRL+z (shell)  Sends a STOP signal (bg) ● CTRL+c (shell)  Aborts the execution – ls /proc  Numbers are currently running processes Master on Free Software
  • Index ● Basic commands – Files & directories – Process control – Documentation – Redirections & pipes Master on Free Software
  • Documentation ● program ­­help  Shows parameters and a brief summary of the program. Given by the own program. ● man [command|function]  Show online help for the program or command ● apropos word  Show commands related to the given word Master on Free Software
  • Documentation ● info program  Shows extended information about a program – Browseable text format – Organized in a different way than man ● /usr/doc/package /usr/share/doc/package  Specific doc about each package: license, usage examples, etc. Master on Free Software
  • Index ● Basic commands – Files & directories – Process control – Documentation – Redirections & pipes Master on Free Software
  • Shell: redirections & pipes ● In Unix, the shell was traditionally sh. In GNU/Linux: bash, ksh, csh, dash,... ● Each process has at least three I/O ways: – stdin (0): Standard input, usually bound to the current terminal (keyboard) – stdout (1): Standard output, usually bound to the terminal (screen) – stderr (2): Standard error, bound to the terminal (screen) together with stdout Master on Free Software
  • Shell: redirections & pipes ● Output redirection – prog > file  Overwrites the output file – prog >> file  Appends to output file ● Input redirection – prog < file  Reads input from file ● Error redirection – prog 2> file  Overwrites the output file – prog 2>> file  Appends Master on Free Software
  • Shell: redirections & pipes ● Examples: $ echo Hello > out.txt $ wc < out.txt $ echo Hello again >> out.txt 1 3 14 $ cat out.txt Hello $ wc xyz.txt Hello again wc: xyz.txt: File not found $ echo See you > out.txt $ wc xyz.txt 2> err.txt $ echo See you again > out.txt $ cat err.txt $ cat out.txt wc: xyz.txt: File not found See you again Master on Free Software
  • Shell: redirections & pipes ● Pipes: One program sends data through a pipe and another one reads it from the opposite side ● prog1|prog2 ● Examples: $ echo See you later | wc 1 3 14 $ tar zcvf ­ . | nc 192.168.1.2 10000 $ nc ­l ­p 10000 | tar zxvf ­ Master on Free Software
  • Script programming Master on Free Software
  • Index ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software
  • Index ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software
  • Defining env. variables ● Variable declaration Quotes aren't mandatory, VARIABLE=”value” but recommended ● Variable usage (substitution for its Don't type content) spaces $VARIABLE   ${VARIABLE} ● Variable deletion unset VARIABLE Master on Free Software
  • Defining env. variables ● Examples: $ S=”_s_”; SEA=”_sea_” $ echo $SEA; echo ${S}EA _sea_ _s_EA Master on Free Software
  • Defining env. variables ● Variable substitution ● $param, ${param}  Simple substitution (for the value of param) ● ${param:­value}  If param is null, use value ● ${param:=value}  If param is null, use value and assign param=value Master on Free Software
  • Defining env. variables ● Variable substitution ● ${param:?value}  If param is null, write value to stderr. If value isn't specified, writes “param: param null or not set” ● ${param:+value}  If param is defined, use value. If not, don't substitute i man bash (“EXPANSION, Parameter expansion” section) Master on Free Software
  • Defining env. variables ● Examples: $ echo “Name is ${NAME:­unknown}” Name is unknown $ NAME=Peter; echo “Name is ${NAME:­unknown}” Name is Peter $ echo “Before: ${N}, Now: ${N:='Peter'}, After: ${N}” Before: , Now: 'Peter', After: 'Peter' $ unset NAME; echo “Name is ${NAME:?}” bash: NAME: parameter null or not set $ NAME='Peter'; echo “${NAME:+Exists}” Exists Master on Free Software
  • Defining env. variables ● Data read to be stored in variables read A B C... – Input fields are usually split by spaces, line feeds or tabs – This behavior can be modified changing the IFS environment variable (internal field separator) – Example: $ export IFS=”:”; read A B C; echo “A/B/C” 1:new york;z77c6s 1/new york/z77c6s Master on Free Software
  • Global variables, Language ● Variables assigned in the current shell (locals) aren't inherited by other child shells. They must be exported export DISPLAY='localhost:0.0' ● Read only variables: readonly VARIABLE ● Examples: $ NAME=Peter $ NAME=John $ readonly NAME $ NAME=Peter bash: NAME: readonly variable Master on Free Software
  • Global variables, Language ● List of defined variables: set ● Most important global variables – DISPLAY  X-Window server which the current process will send windows to – COLUMNS  Number of console columns – EDITOR  Favourite text editor – EUID  Current effective User ID – HOME  Path to the user home directory Master on Free Software
  • Global variables, Language ● Most important global variables – HOSTNAME  Computer name – HOSTTYPE  Computer architecture – IFS  Internal field separator, input field separator (parameters, etc.) – LINES  Number of console lines – OSTYPE  Operating system – PAGER  Favourite pager (eg: more, less, most) Master on Free Software
  • Global variables, Language ● Most important global variables – PATH  Executable files search path. Split by “:” – Example: PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin – PPID  Parent Process ID – PS1  Primary prompt string. Supports substitutions i man bash (“PROMPTING” section) Master on Free Software
  • Global variables, Language ● Most important global variables – PS2  Secondary prompt string (when some lines are used to type a command). Eg: cutting with “” – PS3  Prompt string for select command – PS4  Nesting indicator in debug mode – PWD  Current work path – SHELL  Current command interpreter Master on Free Software
  • Global variables, Language ● Most important global variables – TERM  Terminal emulator. Eg: linux, vt100, xterm – UID  Current User ID – USER  Current user i man bash (“PARAMETERS, Shell variables” section) Master on Free Software
  • Global variables, Language ● Language related variables – LANG  Locale (lang. localization) to be used. ● Eg: es_ES, es_ES@euro, es_GL, es_ES.UTF-8 – LC_ALL  Has preference over LANG and any other LC_ variable to choose a localization – LC_COLLATE  Preference order for path sorting, range expressions, etc. – LC_TYPE  Character interpretation and character class behavior Master on Free Software
  • Global variables, Language ● Language related variables – LC_MESSAGES  Translation for system messages and any other string like $”message” – LC_NUMERIC  Numeric format for a particular localization – LC_TIME  Date and time format i man locale Master on Free Software
  • Global variables, Language ● Shell special variables – $0...$9  Shell executable name and 9 first arguments that were passed – $#  Number of args. passed to the shell – $*  String with all the args. as one single value – $@  Like $*, but when wrapped by double quotes, wraps each parameter by double quotes Master on Free Software
  • Global variables, Language ● Shell special variables – $$  Current script or session PID – $!  PID of the program last sent to background – $?  Exit status of the last program not executed in background – $­  Current options in effect – ~  Path to the user home directory i man bash (“PARAMETERS, Special parameters” section) Master on Free Software
  • Index ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software
  • Input/output ● Input/output – >, <  Stdin and stdout redirection – |  Pipes, redirection of a command stdout to the stdin of the next one – <<  Read stdin from the script itself until a mark is found (here documents) $ cat << END > Hello, I'm $USER. > Good afternoon. END Hello, I'm root. Good afternoon. Master on Free Software
  • Input/output ● Input/output – >>  Stdout redirection in append mode – 0<file, 0<<file  Redirection of the input of a descriptor (0) to a file – 2>file, 2>>file  Redirection of the output of a descriptor (2) to a file – 0<&5  Redirection of the input of a descriptor (0) to the file used by another descriptor (5) Master on Free Software
  • Input/output ● Input/output – 2>&1  Redirection of the output of a descriptor (2) to the file used by another descriptor – <&­  Closes stdin – >&­  Closes stdout – script>>file  Redirection to a file of the stdout for all the output of the commands of a script Master on Free Software
  • Input/output ● Input/output – for ... done >> file  Redirection to a file of the stdout for all the operation of a loop ● Interesting files for input/output – /dev/null  Discards everything sent to it. Useful to avoid displaying a command output – /dev/tty  Our own terminal. Anything sent to it, is displayed on the screen Master on Free Software
  • Index ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software
  • Command grouping ● Command grouping – Semicolon (;)  Chains commands in the same line as if they were a single one – Parenthesis ( )  Executes the code inside in a subshell. Changes in variables don't stay after returning to the current shell $ TMP=stays; (TMP=you_cant_see_me; echo $TMP); echo $TMP Typing spaces and ending in ; isn't mandatory Master on Free Software
  • Command grouping ● Command grouping – Curly braces { }  Executes the code inside in the current shell $ TMP=stays; { TMP=you_cant_see_me; echo $TMP; }; echo $TMP Typing spaces and ending in ; is mandatory – Double ampersand (&&)  Executes the next command only if the previous one returned a true result (equal to 0) $ ls mydir && echo 'That was the content of mydir' Master on Free Software
  • Command grouping ● Command grouping – Double pipe (||)  Executes the next command only if the previous one returned a false result (different from 0) $ ls mydir || echo 'The directory mydir doesn't exist' Master on Free Software
  • Index ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software
  • Wildcards ● File name matching in the shell: wildcards – ?  Matches with one single character – *  Matches zero or more characters – [abc...]  Matches with any of the listed characters – [!abc...]  Matches with all but the listed characters Master on Free Software
  • Wildcards ● File name matching in the shell: wildcards – {str1,str2,...}  Makes a list of expressions using one string each time file.{txt,pdf,rtf}   file.txt file.pdf file.rtf – (escape)  Overrides the meaning of any special character, even end of line Master on Free Software
  • Index ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software
  • Quoting ● Double quotes (“”)  Overrides the special meaning of all the characters but $, ` and $ echo “The user is $USER” The user is root ● Single quotes ('')  Overrides the special meaning of all the characters $ echo 'The user is $USER' The user is $USER Master on Free Software
  • Quoting ● Back quotes (``)  Command substitution. It's executed and output is put in place of the command text $ echo “Today is ###`date`###” Today is ###Thu Jan 10 11:52:26 CET 2008### ● Backslash ()  Overrides the special meaning of the next character, even inside quotes $ echo “Double quotes ”are nice”, aren't they?” Double quotes “are nice”, aren't they? Master on Free Software
  • Index ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software
  • Control structures ● Condition evaluation – test 5 = 5 These two commands are the same (see /usr/bin). It's important to keep the space. – [ 5 = 5 ] 5 = 5 are in fact command arguments ● The result is stored in $? (errorlevel) – True: 0 – False: different from 0 Master on Free Software
  • Control structures ● Possible test conditions – string  string isn't null – ­n string  string isn't null – string1 = string2  Are equal – string1 != string2  Are distinct – ­eq  Numerically equal to (put between the values to be compared) – ­ne  Not equal Master on Free Software
  • Control structures ● Possible test conditions – ­gt  Greater than – ­ge  Greater or equal – ­lt  Lower than – ­le  Lower or equal – ­b  File is a special block file – ­c  File is a special character file – ­d  Is a directory Master on Free Software
  • Control structures ● Possible test conditions – ­f  Is an ordinary file – ­g  Has the setgid bit active – ­k  Has the sticky bit active – ­p  Is a named pipe – ­r  Can be read by the current process – ­s  Size larger than 0 – ­t  File descriptor is open and linked to a terminal Master on Free Software
  • Control structures ● Possible test conditions – ­u  Has setuid bit active – ­w  Can be written by current process – ­x  Can be executed by current process – ­a  AND of previous and next arguments – ­o  OR of previous and next arguments i man test Master on Free Software
  • Index ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software
  • Control structures ● If conditional structure if [ $n ­lt 5 ] then  echo 'lower than' elseif [ $n ­gt 5 ] then  echo 'greater than' else  echo 'are...'  echo '...equal' fi Master on Free Software
  • Control structures ● Case conditional structure case $option in  ­h) echo 'This script has no help'  ;;  ­n) echo 'The computer name is:'  hostname  ;;  *) echo 'Default option'  ;; esac Master on Free Software
  • Control structures ● For loop (list) for variable in 1 2 3 /tmp/* `ls` list... do  echo 'One element:' List formed by single elements, files in a path (even with wildcards  echo $variable expanded to multiple values), done output of command execution, etc. ● If for parameters are skipped, positional parameters $1..$9 are used Master on Free Software
  • Control structures ● For loop (counter) for ((i=0;$i<5;i++)) do  echo 'Number:' Value change  echo $i Initialization done Condition Master on Free Software
  • Control structures ● While loop i=1 while [ $i ­le 5 ] do  echo $i  i=`expr $i + 1` done Master on Free Software
  • Control structures ● Until loop i=1 until [ $i ­gt 5 ] do  echo $i  i=`expr $i + 1` done ● Iteration control in loops – break  Exits the loop immediately – continue  Jumps to the next iteration Master on Free Software
  • Control structures ● Functions The space between the function and () isn't mandatory, nor even the – Declaration { appearing in the same line. Parameters are not declared. union () {  echo 'Joining the arguments'  return “$1 union $2” Parameters appear as positional } parameters anotherFunction () { echo 'hello'; } ● Usage union “one” “two” Result is declared by return and echo $? used with $? Master on Free Software
  • Control structures ● Functions – Global scope of the variables: variables declared or changed inside a function stay when the call ends ● Code inclusion: dot (.) #!/bin/sh . utilities.sh myUsefulFunction 1 2 3 4 Master on Free Software
  • Code examples ● Parameter shifting #!/bin/sh # FILE printargs.sh while [ $# ­gt 0 ] do  echo “$@” Shifts the positional parameters  shift done deleting the first one ($1) $ printargs.sh fly from “New York” to Moscow  fly from New York to Moscow  from New York to Moscow  New York to Moscow  to Moscow  Moscow Master on Free Software
  • Code examples ● Command line parameter processing (using shift) #!/bin/sh # setether: set an Ethernet interface's IP config while [ $# ­gt 1 ] do  case $1 in Shifts two positional  a) ARP=”arp”; shift;;   parameters in a row  b) BROADCAST=$2; shift 2;;  i) IPADDRESS=$2; shift 2;;   m) NETMASK=$2; shift 2;;  n) NETWORK=$2; shift 2;;  *) echo “setether: illegal option: $1”; exit 1;;  esac done INTERFACE=$1 ifconfig $INTERFACE $IPADDRESS netmask $NETMASK   broadcast $BROADCAST $ARP; route add ­net $NETWORK Master on Free Software
  • Code examples ● Command line parameter processing (using getopts) #!/bin/sh # setether: set an Ethernet interface's IP config while getopts ab:e:i:m:n: option do  case $1 in  a) ARP=”arp”;;  b) BROADCAST=$OPTARG;; The : indicates that this  i) IPADDRESS=$OPTARG;; parameter takes an argument  m) NETMASK=$OPTARG;;  n) NETWORK=$OPTARG;;  *) echo “setether: illegal option: $option”; exit 1;;  esac done INTERFACE=$1 ifconfig $INTERFACE $IPADDRESS netmask $NETMASK   broadcast $BROADCAST $ARP; route add ­net $NETWORK Master on Free Software
  • Advanced commands Master on Free Software
  • Index ● Advanced commands – Regular expressions – Grep – Sed – Awk Master on Free Software
  • Index ● Advanced commands – Regular expressions – Grep – Sed – Awk Master on Free Software
  • Regular expressions ● Regex: Powerful text pattern matching expressions ● Use metacharacters to mean repetition, existence or ranges in character patterns ● Common metacharacters – Dot (.)  Matches with any character but line feed – Asterisk (*)  Matches 0 or more appearings of the char. or regex preceeding it Master on Free Software
  • Regular expressions ● Common metacharacters – Brackets []  Match any of the char. classes enclosed by them ●Circumflex (^) after the opening bracket  Inverse match ●Minus (­)  Specifies a range ● Metacharacters loose their meaning inside the brackets. Eg: [ at the begining of the list Master on Free Software
  • Regular expressions ● Common metacharacters – Circumflex (^)  Means line begining – Dollar ($)  Means line ending – Backslash ()  Escape next metacharacter ● Example # grep '^.* 15:[0­9][0­9].*kernel.*$' /var/log/messages Sep 17 15:04:13 localhost kernel: eth1: remaining active... Sep 17 15:04:18 localhost kernel: eth1: DSPCFG accepted after... Sep 17 15:04:18 localhost kernel: eth1: link up. ... Master on Free Software
  • Regular expressions ● Metacharacters exclusive to sed, grep, egrep and awk – Slash-brace (/{m,n/})  Matches a range of appearings of the single character immediately before the expression – Variants ● /{m/}  Exactly m repetitions ● /{m,/}  At least m repetitions ● /{m,n/}  Betwee m and n repetitions Master on Free Software
  • Regular expressions ● Metacharacters exclusive to sed, grep, egrep and awk – Plus (+)  Matches one or more appearings of the character or regex before it – Question mark (?)  0 or 1 appearings – Pipe (|)  Matches the previous expression or the next one – Parenthesis ()  Groups regular expressions Master on Free Software
  • Regular expressions i man -S7 regex ● Tip: Remember that some characters are interpreted by the shell (hello|hi) world          (hello|hi) world  '(hello|hi) world' ● Regex learning & exploring utilities: – kodos – kregexpeditor – visual-regexp Master on Free Software
  • Index ● Advanced commands – Regular expressions – Grep – Sed – Awk Master on Free Software
  • Grep ● Grep – Outputs the lines from stdin or file(s) matching a pattern – Only supports basic regexps – Example: Modified files in a local repository svnstatus | grep ­v nbproject | grep ^M Inverse matching Master on Free Software
  • Grep ● Egrep – Supports extended syntax ● Rgrep – Supports recursive search – Use -E for extended syntax Master on Free Software
  • Index ● Advanced commands – Regular expressions – Grep – Sed – Awk Master on Free Software
  • Sed ● Sed: Stream editor for filtering and transforming text sed ­e script [file] sed ­f scriptfile [file] ● Substitution s/source/destination/modifiers – Example: for SRC in `find ­type f` do  DEST=`echo $SRC | sed ­e 's/.txt/.bak/'`  mv $SRC $DEST done Master on Free Software
  • Sed ● Substitution – Useful modifiers ● g  Global (multiple) replacement per line sed ­e 's/./,/g' ● w file  Write pattern to file sed ­e 's/kernel/&/w kernel.txt'  ­e 's/gconf/&/w gconf.txt' < /var/log/syslog ● p / !p  Print/hide substitued lines (use ­n to hide default output). ! inverts the restriction ls ­lisa | sed ­n ­e 's/total/total/p' Master on Free Software
  • Sed ● Substitution – Using parts of the result ● Whole result (&) ● Substrings: (, ), 1, 2... sed ­e 's/([0­9]*).([0­9]*)/Integer: 1, decimal: 2/' 3452345.46657 Integer part: 3452345, decimal part: 46657 Master on Free Software
  • Sed ● Patterns and ranges to apply commands – Line: sed ­n ­e '12 s/src/dest' – Range of lines: sed ­n ­e '12,25 s/src/dest' sed ­n ­e '1,25 p' sed ­n ­e '12,$ s/src/dest/!p' Master on Free Software
  • Sed ● Patterns and ranges to apply commands – Matching a pattern: sed ­n ­e '/pattern/ s/src/dest' – Matching a range between pattern occurences: sed ­n ­e '/begin/,/end/ p' Master on Free Software
  • Sed ● Command grouping { } sed ­n ­e '  /begin/,/end/ { When the pattern is   s/1/A/gp matched, the block is executed.   s/2/B/gp When not matched, the general command is   s/3/C/gp executed.  }  p ' Master on Free Software
  • Sed ● More advanced commands – a  Append lines – i  Insert lines – c  Change lines – =  Print line number i man sed http://www.grymoire.com/Unix/Sed.html http://www.panix.com/~elflord/unix/sed.html Master on Free Software
  • Index ● Advanced commands – Regular expressions – Grep – Sed – Awk Master on Free Software
  • Awk ● Awk: pattern scanning and processing language awk program [file] awk ­f programfile [file] ● Most common syntax – Line matching awk /regex/ file Master on Free Software
  • Awk ● Most common syntax i-th word (like if it was a parameter) in the matched – Matching and processing line. $0 is the whole line. awk '  BEGIN    {print “Initialization”}  /regex1/ {print $1, $2}  /regex2/ {print $3, $4}  END      {print “Final action”} ' file – Master on Free Software
  • Awk ● Most common syntax – Programming structures inside code blocks { if ($1==”1”) print “Winner”; else print “Looser”; } { x=1; while (x<100) { x=x*2; } } { for (i=0;i<10;i++) { print i; } Master on Free Software
  • Awk ● Most common syntax – Predefined variables (most important) ● NR  Current count of number of input lines ● NF  Count of number of words in current line ● FILENAME  Current processed file name i man awk http://www.vectorsite.net/tsawk.html Master on Free Software