System integration in free software

Systems Integration on Free Software Enrique Ocaña González Master on Free Software

Index (1/4) ● Administrative tasks – System startup – Users & groups – Task automating – Auditing – Backups Master on Free Software

Index (2/4) ● Basic commands – Files & directories – Process control – Documentation – Redirections & pipes Master on Free Software

Index (3/4) ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software

Index (4/4) ● Advanced commands – Regular expressions – Grep – Sed – Awk Master on Free Software

Administrative tasks Master on Free Software

Index ● System startup ● Users & groups ● Task automating ● Auditing ● Backups Master on Free Software

Introduction ● Administration: security – Two roles ● User: Not concerned about administrative tasks ● Administrator (root): Responsible for the proper operation of the system – System designed thinking in program and user data separation – Advice: Only work as root when needed ● Avoids unintended damage of the system Master on Free Software

Introduction ● Administration: transparency – Config and utilities based on plain text files and scripts – Looking at the configuration, reading documentation and following what is done by the scripts, it's possible to understand what the system does – The root of a problem can be found in a simpler way than in closed systems, where the internal system operation isn't known Master on Free Software

Introduction ● Administration: support – User community at internet – Specialized companies Master on Free Software

Introduction ● Steps to solve a problem – Read documentation ● /usr/share/doc/package ● Project website ● FAQs and howtos – Upgrade the program to the latest version (maybe the bugs have been solved) – Get in touch with the developers – Return the favour colaborating in projects ;-) Master on Free Software

System startup ● System startup sequence: 1.BIOS load, POST tasks, boot sequence selection 2.Boot sector load (MBR) 3.Boot loader: grub, lilo, ntldr 4.Kernel load: /boot/vmlinuz 5. init program execution 6.System tasks: /etc/init.d/rcS 7.Runlevel specific tasks: /etc/rcN.d Master on Free Software

System shutdown ● System shutdown – Init process receives a state change signal that lets stopping (runlevel 0) or restarting (runlevel 6) the system ● shutdown ● halt ● init – Executes the actions associated to the new runlevel in a stablished order Master on Free Software

System shutdown ● System shutdown stages: 1. Shutdown warning to users and services 2. Execution of actions associated to the system shutdown (runlevel change) 3. Proper filesystem unmounting 4. Computer power off or reset Master on Free Software

Init and runlevels ● Init process: – Father or all the processes (PID 1) – Has different states or runlevels with services and actions associated, to be executed when entering into the runlevel or leaving it – The most common runlevels are: ● 0  System shutdown ● 6  System restart ● 1  Single user mode ● 2-5  Multiuser (2 by default) Master on Free Software

Init and runlevels ● Configuration files: – Master init configuration: /etc/event.d ● Used to be at: /etc/inittab – Real init scripts placed at /etc/init.d – System bootstrap init links: /etc/rcS.d – Runlevel links: /etc/rcN.d (N=0...6) Master on Free Software

Init and runlevels ● Symbolic links format: – S|K + order + service name ● Eg. at rc2.d: S89cron > ../init.d/cron ● S  Start the service when entering the runlevel ● K  Kill the service when entering the runlevel ● Manual startup: – /etc/init.d/service {start|stop| restart|reload|forcereload} – /etc/init.d/service shows help Master on Free Software

Boot manager ● Common boot managers: – lilo – grub i Additional documentation: boot_loader_config.pdf Master on Free Software

Users & groups ● root: the user with the highest privileges ● Groups allow granting and grouping privileges of a set of users ● UID, GID: Unique numerical identifiers – root has UID 0 ● Base directory or “home” – /home/user – /root Master on Free Software

Users & groups ● Files involved: – /etc/passwd: Information about the user accounts ● Login ● Password – Crypted, “x” when using shadow passwords, “!” when the account is disabled ● UID ● Home directory path ● Shell ● Name, surname, other identification info Master on Free Software

Users & groups ● Files involved: – /etc/shadow: Like passwd, but only readable by root and meant to store the passwords – /etc/group: Info about groups and which users belong to each one – /etc/gshadow: Non public info about the groups Master on Free Software

User management ● User suplantation su user ● User task suplantation (“run as”) sudo user command ● Suplantation access control – File /etc/sudoers (don't touch!) – visudo Master on Free Software

User management ● User creation – adduser user ● Creation process 1. Entry in passwd file 2. Entry in shadow file 3. Creation of /home/user 4. Copy of the files in /etc/skel 5. Password setting Master on Free Software

User management ● User deletion – deluser user ● Creation process 1. Delete entries in passwd and shadow files 2. Finds all the files outside the home directory and deletes them, making a security copy Master on Free Software

Group management ● Adding a group – addgroup group ● Deleting a group – delgroup group ● Process: Adds/deletes the entries in the files /etc/group and /etc/gshadow Master on Free Software

Group management ● Adding a user to an existing group – adduser user group ● Deleting a group – adduser user group ● Process: Adds/deletes the entries in the files /etc/group and /etc/gshadow Master on Free Software

Password changing ● Change the password of a user – passwd  Changes the password of the current user – passwd user group  Executed as root, changes the password of the user specified ● Executed by a normal user, the command asks for the old password ● Executed by root, it doesn't Master on Free Software

User monitoring ● Each time a user logs into the system, at some logs are recorded: – /var/log/syslog  Generic log for all the system events. Maybe also logged on auth. – /var/log/wtmp  Specific log for all the user logins in the system. ● Binary file – /var/log/btmp  Specific log for all the failed user logins in the system. Master on Free Software

User monitoring ● Commands – who (or w)  Shows who is currently logged into the system and what are they doing – last / lastb  All the users that logged into the system recently. Lastb shows info of failed attempts – slay  Kicks a user out of the system and kills all her processes Master on Free Software

Task automating ● cron  System utility that allows launching periodic processes (tasks, actions...) in an unattended way, at a scheduled date ● at  System utility that allows to program the execution of a process, but without repetition. It allows to defer the unattended process until a date Master on Free Software

Repeated tasks: cron ● The cron service consists of 2 programs: – crond  Program (daemon) offering the service to the system – crontab  Program that allows an user to manage her tasks ● crontab e  Edit the scheduled tasks of the current user ● crontab l  List the tasks of the current user ● crontab r  Remove all the tasks of the current user Master on Free Software

Repeated tasks: cron ● Crontab edition: – crontab e  Opens an editor predefined by the variable EDITOR, to edit the user crontab ● Tip: export EDITOR=nano – Each crontab line has the following format, always split by spaces ● Minute: Integer number between 0 and 59 ● Hour: Integer in the range 0-23 ● Day: Integer in the range 1-31 (must be a valid day if the month is also specified) Master on Free Software

Repeated tasks: cron ● Crontab edition: ● Month: Integer in the range 1-12 ● Week_day: Integer in the range 0-7, where 0 and 7 mean Sunday. The short name of the week day can also be used (mon, tue, wed...) ● Command: Command to be executed. The command can be something like “ls /proc >> /tmp/proc” or a command that executes a customized script written by the user Master on Free Software

Repeated tasks: cron ● Moreover... – A “*” in any field indicates all valid values for that field (respecting the restricions of the rest of the fields) – A “–“ between numbers means an interval between those numbers – A list of values split by “,” means a value list – A “/” means step values. Eg: “*/15” would mean that the task is executed each 15 minutes. Master on Free Software

Repeated tasks: cron ● Moreover... – Lines starting with “#” are comments and aren't had into account – A task is executed in the moment when all the requirements of the schedule parameters are met Master on Free Software

Repeated tasks: cron ● Moreover... – Special parameters (instead of the 5 said fields) ● @reboot  One time when the computer boots ● @yearly  Once a year, eq. to 0 0 1 1 * ● @daily  Each day, eq. to 0 0 * * * ● @weekly  Each week, eq. to 0 0 * * 0 ● @monthly  Each month, eq. to 0 0 1 * * ● @hourly  Each hour, eq. to 0 * * * * Master on Free Software

Repeated tasks: cron ● Examples (crontab -e) # min hour day month weekday command # Run 5 min after midnight, every day 5 0 * * * /bin/daily.sh >> /tmp/out 2>&1 # Run at 2:15pm on the first of every month 15 14 1 * * $HOME/bin/monthly # Run at 10pm on weekdays 0 22 * * 15 /usr/local/bin/script.sh # Run 23 min after midnight, 2am, 4am... everyday 23 023/2 * * * script.sh Master on Free Software

Delayed tasks: at ● at allows to defer the execution of a task. Usage: at date task CTRL+d ● Example: $ at 4:45pm at> echo “Time to go” CTRL+d Master on Free Software

Delayed tasks: at ● atq  Shows pending tasks # atq 1 20080101 10:35 b peter 2 20080215 00:00 c peter 5 20080210 01:00 Z root ● at -c 2  Shows all the execution environment for task 2 ● atrm tasknum  Deletes the specified task from the task queue Master on Free Software

Delayed tasks: at ● Considerations – Admin can control which users cat use at: /etc/at.allow, at.deny. She can also see the tasks of all users – at has a queue system to set the priority of the tasks – batch is a command derived form at that allows executing a task only when the system load goes below of 0.8 Master on Free Software

Log location and format ● At log files are located at /var/log ● Important files – messages  System wide messages – syslog  Messages gathered by the log daemon (syslogd) – debug  System logs useful to solve problems – auth.log  Login and logoff attempts Master on Free Software

Log location and format ● Log format Oct 26 18:17:01 localhost CRON[3384]: (pam_unix) session opened... Log line Process name and PID Computer Date and time ● Log continuous viewing tail f logfile Master on Free Software

Log location and format ● Log files with a non-standard format – dmesg  Last system startup info – faillog  Info about the last time users attempted to log into the system (fail number, max allowed, last fail). ● Viewed using faillog u user – lastlog  List of users and the last time they logged into the system. ● Viewed using lastlog Master on Free Software

Log location and format ● Log files with a non-standard format – Xfree86.0.log, Xorg.0.log  Info from the X-Window system. ● Viewed with standard viewers (tail, etc.) i http://www.debian.org/doc/manuals/users-guide/ch-rsi.en.html Master on Free Software

Log analysis and tools ● Log analysis allows to detect intrusions in the computer and hardware/software problems ● logcheck  Looks for strange log entries and sends them to the administrator via email i logcheck.pdf Master on Free Software

Log configuration ● Log file configuration: /etc/syslog.conf ● Format: – Selector: service.priority [[, service...]; service.priority...] ● Services: auth, authpriv, cron, daemon, ftp, kern, lpr, news, syslog, user, uucp, local0...7, *, none ● Priorities: debug, info, notice, warning, crit, alert, emerg, *, none – =priority: only that priority, not higher ones – !priority: only that priority, not lower ones – !=priority: exactly that priority Master on Free Software

Log configuration ● Format: – Action ● /.../file: Log to log file (use complete path) ● |pipe: Log to named pipe ● /dev/console: Log to the console ● @hostname: Log to a remote computer ● user1, user2, ...: Send to a list of users ● *: Send to all the users Master on Free Software

Log configuration ● Example: kern.* /var/adm/kernel kern.crit @earth kern.crit /dev/console kern.info;kern.!err /var/adm/kernelinfo mail.*;mail.!=info /var/adm/mail *.* @mars i man syslog.conf, man syslogd Master on Free Software

Log configuration ● Log rotation: logs are rotated periodically. They are compressed and deleted by age order ● Example: – /var/log/syslog – /var/log/syslog.0 – /var/log/syslog.1.gz – /var/log/syslog.2.gz Master on Free Software

Log configuration ● Configuración ● /etc/logrotate.conf: Main log rotation configuration ● /etc/logrotate.d/*: Specific configuration for particular services i man logrotate Master on Free Software

Backup policy ● Backup levels – Level 0: Complete backup – Level 1: Incremental, changes from level 0 – Levels 2..9: Changes from previous level ● Storage media – Tape – CD, DVD – Hard disk – Network Master on Free Software

Backup policy ● Backup calendar – Diary complete Backup level M T W T F S S 0 0 0 0 0 0 0 – Weekly complete, diary level 1 M T W T F S S 1 1 1 1 1 1 0 – Weekly complete, diary level 1 M T W T F S S 1 2 3 4 5 6 0 Master on Free Software

Backup policy ● Backup calendar – Monthly hanoi tower M T W T F S S 3 2 5 4 7 6 1 3 2 5 4 7 6 1 3 2 5 4 7 6 1 3 2 5 4 7 6 0 i http://www.backupschedule.net/backupSchedules/towerOfHanoi.html Master on Free Software

Automating and scripts ● Ad-hoc backup: tar ● Saving tar zcvf file.tgz files... gzip create verbose file ● Restoring tar zxvf file.tgz [files...] extract Master on Free Software

Automating and scripts ● Ad-hoc backup: tar – Other functions ● A: Concatenate some tar files (not tgz!) ● d: Find differences between tar/tgz file and files on disk ● delete: Delete files in tar/tgz file ● r: Append files at the end of a tar/tgz file ● t: List contents ● u: Only append files newer than the ones in the tar/tgz file Master on Free Software

Automating and scripts ● Other useful commands – cpio: Input/output copy to file find . print depth | cpio ov > tree.cpio cpio idv < tree.cpio – rsync: Remote directory synchronization rsync aP e ssh user@host:rem_path loc_path – dd: Exact copy from one device to another (or to a file) dd if=/dev/hda1 of=bak_hda1.bin Master on Free Software

Automating and scripts ● Other useful commands – mt: Magnetic tape management mt f /dev/st0 eject mt retension mt status – Example of an ad-hoc backup on tape #!/bin/sh # SYSTEM CONFIG AND DATA BACKUP tar jcvlf exclude=/var/www /var /etc \\ | buffer B s 10k > /dev/st0 mt offline Master on Free Software

Automating and scripts ● Simple automating: cron ● Advanced backup utilities: Amanda – Backup automating – Network backup server, multiple client platforms (even Windows using Samba) – Mail reminder for administrators – Tape dump – Intelligent estimation of backup leven based on previous statistics Master on Free Software

Automating and scripts ● Amanda configuration – Multiple configs. Eg: /etc/amanda/DailySet1 – Authorized clients: /etc/amandahosts – Main config: amanda.conf – List of disks to be dumped: disklist – List of tapes to be used: tapelist (don't touch!) Master on Free Software

Automating and scripts ● Tape labeling for Amanda use amlabel [f] DailySet1 DMP00 Force, for a Configuration to Tape identifier (syntax defined previously be used at amanda.conf) labeled tape Master on Free Software

Automating and scripts ● Recovering: amrecover root@cesar:~ # cd / root@cesar:/ # amrecover C DailySet1 s localhost.localdomain AMRECOVER Version 2.4.4p3. Contacting server on localhost ... The files are going to be extracted in the 220 localhost.localdomain AMANDA index server (2.4.4p3) ready. 200 Access OK current directory. It's important to change to Setting restore date to today (20041103) computer to be the root directory (“/”) of the restored, in particular if it's a satellite 200 Working date set to 20041103. ... computer (not the one hosting the backup) amrecover> listdisk 200 List of disk for host localhost.localdomain 201 /home 201 /etc 201 /var 200 List of disk for host localhost.localdomain amrecover> setdisk /var 200 Disk set to /var. Master on Free Software

Automating and scripts ● Recovering: amrecover root@cesar:~ # cd / root@cesar:/ # amrecover C DailySet1 s localhost.localdomain AMRECOVER Version 2.4.4p3. Contacting server on localhost ... 220 localhost.localdomain AMANDA index server (2.4.4p3) ready. 200 Access OK Setting restore date to today (20041103) NOTE: check access 200 Working date set to 20041103. permissions at ... /etc/amandahosts amrecover> listdisk 200 List of disk for host localhost.localdomain 201 /home 201 /etc 201 /var 200 List of disk for host localhost.localdomain amrecover> setdisk /var 200 Disk set to /var. Master on Free Software

Automating and scripts ● Recovering: amrecover amrecover> setdate 20041020 200 Working date set to 20041020. amrecover> ls ... 20041013 spool/ 20041013 www/ amrecover> add www Added dir /www at date 20041013 amrecover> list TAPE DailySet5 LEVEL 0 DATE 20040930 /www TAPE /home/amand... LEVEL 1 DATE 20041013 /www amrecover> extract ... i man amanda , /usr/share/doc/amanda , http://www.amanda.org Master on Free Software

Basic commands Master on Free Software

Index ● Basic commands – Files & directories – Process control – Documentation – Redirections & pipes Master on Free Software

Files basic commands ● I-node: Data structure in a filesystem. Stores basic data about the file, directory or another object in the filesystem – Size and physical location (disk block) – Owner and group – Mode (permissions) – Creation, modification and last access time – Reference counter, accounting how many hard links are pointing to the inode Master on Free Software

Files basic commands ● Example: $ ls lisa total 16 766122 4 drwxrxrx 2 root root 4096 . 765537 4 drwxrwxrwx 11 root root 4096 .. 766125 4 rwrr 2 root root 5 file1 766125 4 rwrr 2 root root 5 file2 766126 0 lrwxrwxrwx 1 root root 8 file3 > file1 Inode Mode Owner and number group Used blocks Reference Used bytes counter Master on Free Software

Files basic commands ● Permissions (mode) – Permissions ● Read (r): 4 (octal) 750 ● Write (w): 2 Others: (no access) ● Execution (x): 1 – Granularity Group: 4(read)+1(execution) ● User (u) User: 4(read)+2(write)+1(execution) ● Group (g) ● Others (o) ● All (a): ugo Master on Free Software

Files basic commands ● Special permissions (4 nibble) th ! UNSAFE!! – Setuid bit (o+s, 4): Sets the owner of the file as the effective user when the file is executed – Setgid bit (g+s, 2): Sets the group of the file as the effective user when executed – Sticky bit (STIcky, Save Text Image) (o+t, 1) ● For files: Keeps the code in swap, just if it's executed later ● For directories: Only the owner of the files inside and the owner of the directory can delete files in the directory Master on Free Software

Files basic commands ● Links – Hard links: The file has two entry points at the same time – Soft links: Reference to the true path of the file a b c d... a b c d... 1234... 1234... Master on Free Software

Files basic commands ● Example $ ls lisa total 16 766122 4 drwxrxrx 2 root root 4096 . 765537 4 drwxrwxrwx 11 root root 4096 .. 766125 4 rwrr 2 root root 5 file1 766125 4 rwrr 2 root root 5 file2 766126 0 lrwxrwxrwx 1 root root 8 file3 > file1 a b c d... a b c d... 1234... 1234... Master on Free Software

Files basic commands ● Example $ ls lisa total 16 766122 4 drwxrxrx 2 root root 4096 . 765537 4 drwxrwxrwx 11 root root 4096 .. 766125 4 rwrr 2 root root 5 file1 766125 4 rwrr 2 root root 5 file2 766126 0 lrwxrwxrwx 1 root root 5 file3 > file1 Reference counter. file1 and file2 are actually two hard links to the same content, thus its reference File the symbolic link is counter is 2 pointing to Byte size (note that “file1” string takes actually 5 bytes) Master on Free Software

Files basic commands ● Example $ ls lisa total 16 766122 4 drwxrxrx 2 root root 4096 . 765537 4 drwxrwxrwx 11 root root 4096 .. 766125 4 rwrr 2 root root 5 file1 766125 4 rwrr 2 root root 5 file2 766126 0 lrwxrwxrwx 1 root root 5 file3 > file1 The current directory (.) has 2 references: Entry (.) of that directory and entry mydirname in its parent directory. Parent directory (..) has 11 references: the entry tmp in the root directory and the “..” entries in the other 10 directories it has. Master on Free Software

Files basic commands ● Special files – Devices (high number, low number) ● Block brwrw 3, 0 /dev/hda ● Character crwrw 9,96 /dev/st0a – Named pipes (FIFO, socketpairs) srwxrwxrwx /tmp/.X11unix/X0 Master on Free Software

Files basic commands ● Special directories: “.”, “..” ● Hidden files: Start by “.”. Eg: .bashrc ● Brief of the file types: d  Directory p  Pipes l  Link s  Sockets c  Char. device  Ordinary b  Block device Master on Free Software

Files basic commands ● pwd  Shows the current absolute path ● cp  Copy files cp [ir] ( file file | dir dir | filelist dir ) ● cd  Change directory cd path ● mv  Move or rename files and directories mv file newfile Master on Free Software

Files basic commands ● mkdir  Create directories mkdir [p] directory ● ls  Shows the content of a directory or info about a file ls [laR] [dirname]|[filename] ● rmdir  Deletes empty directories rmdir directory Master on Free Software

Files basic commands ● rm  Deletes files rm [ir] file1 file2... ● find  Searchs files by name, etc. find [searchdir] name “filename” ● cat  Shows the content of a file without being able to edit it and without pauses cat file Master on Free Software

Files basic commands ● less/more  Shows the content of a file without being able to edit it but page by page less file ● chmod  Permission (mode) change chmod [options] mode file – Examples: ● chmod u+srwx,gw,o+rxt prueba ● chmod 5755 prueba Master on Free Software

Files basic commands ● ln  Link creation – Hard link: Not allowed among directories ln sourcefile destinationlink – Soft link ln s sourcefile destinationlink Master on Free Software

Processes ● Linux is multitasked – Each instance of a command or program being executed is a process ● The kernel controls all the processes – Allocates resources (memory, cpu,...) – Schedules per priority the processes to be executed – Controls resource access. Each process has an owner – Process info at /proc Master on Free Software

Processes ● Important properties characterizing a process – Process identification number (PID) – Parent id number (PPID) – State: Running, Sleep (waiting for a resource) – Owner: Usually the user running the process – Priority: importance in respect to other processes Master on Free Software

Process control commands ● ps  Lists running processes (snapshot) ps aux Shows user, PID, %CPU, %MEM, STAT, time... ● top  Interactively lists running processes. Useful for monitoring. Allows process management ● kill  Sends a termination signal kill KILL pid Causes the abrupt finishing of the process with PID pid Master on Free Software

Process control commands ● nice  Starts process with given priority. Range from -20 (higher) to 19 (lower) nice +10 process ● snice, renice ● bg (shell)  Sends process to background – Same as & when launching from shell ● fg (shell)  Brings a process to foreground. Opposite to bg Master on Free Software

Process control commands ● jobs (shell)  Show processes in background for the current user ● CTRL+z (shell)  Sends a STOP signal (bg) ● CTRL+c (shell)  Aborts the execution – ls /proc  Numbers are currently running processes Master on Free Software

Documentation ● program help  Shows parameters and a brief summary of the program. Given by the own program. ● man [command|function]  Show online help for the program or command ● apropos word  Show commands related to the given word Master on Free Software

Documentation ● info program  Shows extended information about a program – Browseable text format – Organized in a different way than man ● /usr/doc/package /usr/share/doc/package  Specific doc about each package: license, usage examples, etc. Master on Free Software

Shell: redirections & pipes ● In Unix, the shell was traditionally sh. In GNU/Linux: bash, ksh, csh, dash,... ● Each process has at least three I/O ways: – stdin (0): Standard input, usually bound to the current terminal (keyboard) – stdout (1): Standard output, usually bound to the terminal (screen) – stderr (2): Standard error, bound to the terminal (screen) together with stdout Master on Free Software

Shell: redirections & pipes ● Output redirection – prog > file  Overwrites the output file – prog >> file  Appends to output file ● Input redirection – prog < file  Reads input from file ● Error redirection – prog 2> file  Overwrites the output file – prog 2>> file  Appends Master on Free Software

Shell: redirections & pipes ● Examples: $ echo Hello > out.txt $ wc < out.txt $ echo Hello again >> out.txt 1 3 14 $ cat out.txt Hello $ wc xyz.txt Hello again wc: xyz.txt: File not found $ echo See you > out.txt $ wc xyz.txt 2> err.txt $ echo See you again > out.txt $ cat err.txt $ cat out.txt wc: xyz.txt: File not found See you again Master on Free Software

Shell: redirections & pipes ● Pipes: One program sends data through a pipe and another one reads it from the opposite side ● prog1|prog2 ● Examples: $ echo See you later | wc 1 3 14 $ tar zcvf . | nc 192.168.1.2 10000 $ nc l p 10000 | tar zxvf Master on Free Software

Script programming Master on Free Software

Index ● Script programming – Environment variables – Input/output – Command grouping – Wildcards – Quoting – Control structures Master on Free Software

Defining env. variables ● Variable declaration Quotes aren't mandatory, VARIABLE=”value” but recommended ● Variable usage (substitution for its Don't type content) spaces $VARIABLE ${VARIABLE} ● Variable deletion unset VARIABLE Master on Free Software

Defining env. variables ● Examples: $ S=”_s_”; SEA=”_sea_” $ echo $SEA; echo ${S}EA _sea_ _s_EA Master on Free Software

Defining env. variables ● Variable substitution ● $param, ${param}  Simple substitution (for the value of param) ● ${param:value}  If param is null, use value ● ${param:=value}  If param is null, use value and assign param=value Master on Free Software

Defining env. variables ● Variable substitution ● ${param:?value}  If param is null, write value to stderr. If value isn't specified, writes “param: param null or not set” ● ${param:+value}  If param is defined, use value. If not, don't substitute i man bash (“EXPANSION, Parameter expansion” section) Master on Free Software

Defining env. variables ● Examples: $ echo “Name is ${NAME:unknown}” Name is unknown $ NAME=Peter; echo “Name is ${NAME:unknown}” Name is Peter $ echo “Before: ${N}, Now: ${N:='Peter'}, After: ${N}” Before: , Now: 'Peter', After: 'Peter' $ unset NAME; echo “Name is ${NAME:?}” bash: NAME: parameter null or not set $ NAME='Peter'; echo “${NAME:+Exists}” Exists Master on Free Software

Defining env. variables ● Data read to be stored in variables read A B C... – Input fields are usually split by spaces, line feeds or tabs – This behavior can be modified changing the IFS environment variable (internal field separator) – Example: $ export IFS=”:”; read A B C; echo “A/B/C” 1:new york;z77c6s 1/new york/z77c6s Master on Free Software

Global variables, Language ● Variables assigned in the current shell (locals) aren't inherited by other child shells. They must be exported export DISPLAY='localhost:0.0' ● Read only variables: readonly VARIABLE ● Examples: $ NAME=Peter $ NAME=John $ readonly NAME $ NAME=Peter bash: NAME: readonly variable Master on Free Software

Global variables, Language ● List of defined variables: set ● Most important global variables – DISPLAY  X-Window server which the current process will send windows to – COLUMNS  Number of console columns – EDITOR  Favourite text editor – EUID  Current effective User ID – HOME  Path to the user home directory Master on Free Software

Global variables, Language ● Most important global variables – HOSTNAME  Computer name – HOSTTYPE  Computer architecture – IFS  Internal field separator, input field separator (parameters, etc.) – LINES  Number of console lines – OSTYPE  Operating system – PAGER  Favourite pager (eg: more, less, most) Master on Free Software

Global variables, Language ● Most important global variables – PATH  Executable files search path. Split by “:” – Example: PATH=/usr/local/bin:/usr/bin:/bin:/usr/X11R6/bin – PPID  Parent Process ID – PS1  Primary prompt string. Supports substitutions i man bash (“PROMPTING” section) Master on Free Software

Global variables, Language ● Most important global variables – PS2  Secondary prompt string (when some lines are used to type a command). Eg: cutting with “\\” – PS3  Prompt string for select command – PS4  Nesting indicator in debug mode – PWD  Current work path – SHELL  Current command interpreter Master on Free Software

Global variables, Language ● Most important global variables – TERM  Terminal emulator. Eg: linux, vt100, xterm – UID  Current User ID – USER  Current user i man bash (“PARAMETERS, Shell variables” section) Master on Free Software

Global variables, Language ● Language related variables – LANG  Locale (lang. localization) to be used. ● Eg: es_ES, es_ES@euro, es_GL, es_ES.UTF-8 – LC_ALL  Has preference over LANG and any other LC_ variable to choose a localization – LC_COLLATE  Preference order for path sorting, range expressions, etc. – LC_TYPE  Character interpretation and character class behavior Master on Free Software

Global variables, Language ● Language related variables – LC_MESSAGES  Translation for system messages and any other string like $”message” – LC_NUMERIC  Numeric format for a particular localization – LC_TIME  Date and time format i man locale Master on Free Software

Global variables, Language ● Shell special variables – $0...$9  Shell executable name and 9 first arguments that were passed – $#  Number of args. passed to the shell – $*  String with all the args. as one single value – $@  Like $*, but when wrapped by double quotes, wraps each parameter by double quotes Master on Free Software

Global variables, Language ● Shell special variables – $$  Current script or session PID – $!  PID of the program last sent to background – $?  Exit status of the last program not executed in background – $  Current options in effect – ~  Path to the user home directory i man bash (“PARAMETERS, Special parameters” section) Master on Free Software

Input/output ● Input/output – >, <  Stdin and stdout redirection – |  Pipes, redirection of a command stdout to the stdin of the next one – <<  Read stdin from the script itself until a mark is found (here documents) $ cat << END > Hello, I'm $USER. > Good afternoon. END Hello, I'm root. Good afternoon. Master on Free Software

Input/output ● Input/output – >>  Stdout redirection in append mode – 0<file, 0<<file  Redirection of the input of a descriptor (0) to a file – 2>file, 2>>file  Redirection of the output of a descriptor (2) to a file – 0<&5  Redirection of the input of a descriptor (0) to the file used by another descriptor (5) Master on Free Software

Input/output ● Input/output – 2>&1  Redirection of the output of a descriptor (2) to the file used by another descriptor – <&  Closes stdin – >&  Closes stdout – script>>file  Redirection to a file of the stdout for all the output of the commands of a script Master on Free Software

Input/output ● Input/output – for ... done >> file  Redirection to a file of the stdout for all the operation of a loop ● Interesting files for input/output – /dev/null  Discards everything sent to it. Useful to avoid displaying a command output – /dev/tty  Our own terminal. Anything sent to it, is displayed on the screen Master on Free Software

Command grouping ● Command grouping – Semicolon (;)  Chains commands in the same line as if they were a single one – Parenthesis ( )  Executes the code inside in a subshell. Changes in variables don't stay after returning to the current shell $ TMP=stays; (TMP=you_cant_see_me; echo $TMP); echo $TMP Typing spaces and ending in ; isn't mandatory Master on Free Software

Command grouping ● Command grouping – Curly braces { }  Executes the code inside in the current shell $ TMP=stays; { TMP=you_cant_see_me; echo $TMP; }; echo $TMP Typing spaces and ending in ; is mandatory – Double ampersand (&&)  Executes the next command only if the previous one returned a true result (equal to 0) $ ls mydir && echo 'That was the content of mydir' Master on Free Software

Command grouping ● Command grouping – Double pipe (||)  Executes the next command only if the previous one returned a false result (different from 0) $ ls mydir || echo 'The directory mydir doesn\\'t exist' Master on Free Software

Wildcards ● File name matching in the shell: wildcards – ?  Matches with one single character – *  Matches zero or more characters – [abc...]  Matches with any of the listed characters – [!abc...]  Matches with all but the listed characters Master on Free Software

Wildcards ● File name matching in the shell: wildcards – {str1,str2,...}  Makes a list of expressions using one string each time file.{txt,pdf,rtf}  file.txt file.pdf file.rtf – \\ (escape)  Overrides the meaning of any special character, even end of line Master on Free Software

Quoting ● Double quotes (“”)  Overrides the special meaning of all the characters but $, ` and \\ $ echo “The user is $USER” The user is root ● Single quotes ('')  Overrides the special meaning of all the characters $ echo 'The user is $USER' The user is $USER Master on Free Software

Quoting ● Back quotes (``)  Command substitution. It's executed and output is put in place of the command text $ echo “Today is ###`date`###” Today is ###Thu Jan 10 11:52:26 CET 2008### ● Backslash (\\)  Overrides the special meaning of the next character, even inside quotes $ echo “Double quotes \\”are nice\\”, aren't they?” Double quotes “are nice”, aren't they? Master on Free Software

Control structures ● Condition evaluation – test 5 = 5 These two commands are the same (see /usr/bin). It's important to keep the space. – [ 5 = 5 ] 5 = 5 are in fact command arguments ● The result is stored in $? (errorlevel) – True: 0 – False: different from 0 Master on Free Software

Control structures ● Possible test conditions – string  string isn't null – n string  string isn't null – string1 = string2  Are equal – string1 != string2  Are distinct – eq  Numerically equal to (put between the values to be compared) – ne  Not equal Master on Free Software

Control structures ● Possible test conditions – gt  Greater than – ge  Greater or equal – lt  Lower than – le  Lower or equal – b  File is a special block file – c  File is a special character file – d  Is a directory Master on Free Software

Control structures ● Possible test conditions – f  Is an ordinary file – g  Has the setgid bit active – k  Has the sticky bit active – p  Is a named pipe – r  Can be read by the current process – s  Size larger than 0 – t  File descriptor is open and linked to a terminal Master on Free Software

Control structures ● Possible test conditions – u  Has setuid bit active – w  Can be written by current process – x  Can be executed by current process – a  AND of previous and next arguments – o  OR of previous and next arguments i man test Master on Free Software

Control structures ● If conditional structure if [ $n lt 5 ] then echo 'lower than' elseif [ $n gt 5 ] then echo 'greater than' else echo 'are...' echo '...equal' fi Master on Free Software

Control structures ● Case conditional structure case $option in h) echo 'This script has no help' ;; n) echo 'The computer name is:' hostname ;; *) echo 'Default option' ;; esac Master on Free Software

Control structures ● For loop (list) for variable in 1 2 3 /tmp/* `ls` list... do echo 'One element:' List formed by single elements, files in a path (even with wildcards echo $variable expanded to multiple values), done output of command execution, etc. ● If for parameters are skipped, positional parameters $1..$9 are used Master on Free Software

Control structures ● For loop (counter) for ((i=0;$i<5;i++)) do echo 'Number:' Value change echo $i Initialization done Condition Master on Free Software

Control structures ● While loop i=1 while [ $i le 5 ] do echo $i i=`expr $i + 1` done Master on Free Software

Control structures ● Until loop i=1 until [ $i gt 5 ] do echo $i i=`expr $i + 1` done ● Iteration control in loops – break  Exits the loop immediately – continue  Jumps to the next iteration Master on Free Software

Control structures ● Functions The space between the function and () isn't mandatory, nor even the – Declaration { appearing in the same line. Parameters are not declared. union () { echo 'Joining the arguments' return “$1 union $2” Parameters appear as positional } parameters anotherFunction () { echo 'hello'; } ● Usage union “one” “two” Result is declared by return and echo $? used with $? Master on Free Software

Control structures ● Functions – Global scope of the variables: variables declared or changed inside a function stay when the call ends ● Code inclusion: dot (.) #!/bin/sh . utilities.sh myUsefulFunction 1 2 3 4 Master on Free Software

Code examples ● Parameter shifting #!/bin/sh # FILE printargs.sh while [ $# gt 0 ] do echo “$@” Shifts the positional parameters shift done deleting the first one ($1) $ printargs.sh fly from “New York” to Moscow fly from New York to Moscow from New York to Moscow New York to Moscow to Moscow Moscow Master on Free Software

Code examples ● Command line parameter processing (using shift) #!/bin/sh # setether: set an Ethernet interface's IP config while [ $# gt 1 ] do case $1 in Shifts two positional a) ARP=”arp”; shift;; parameters in a row b) BROADCAST=$2; shift 2;; i) IPADDRESS=$2; shift 2;; m) NETMASK=$2; shift 2;; n) NETWORK=$2; shift 2;; *) echo “setether: illegal option: $1”; exit 1;; esac done INTERFACE=$1 ifconfig $INTERFACE $IPADDRESS netmask $NETMASK \\ broadcast $BROADCAST $ARP; route add net $NETWORK Master on Free Software

Code examples ● Command line parameter processing (using getopts) #!/bin/sh # setether: set an Ethernet interface's IP config while getopts ab:e:i:m:n: option do case $1 in a) ARP=”arp”;; b) BROADCAST=$OPTARG;; The : indicates that this i) IPADDRESS=$OPTARG;; parameter takes an argument m) NETMASK=$OPTARG;; n) NETWORK=$OPTARG;; *) echo “setether: illegal option: $option”; exit 1;; esac done INTERFACE=$1 ifconfig $INTERFACE $IPADDRESS netmask $NETMASK \\ broadcast $BROADCAST $ARP; route add net $NETWORK Master on Free Software

Advanced commands Master on Free Software

Index ● Advanced commands – Regular expressions – Grep – Sed – Awk Master on Free Software

Regular expressions ● Regex: Powerful text pattern matching expressions ● Use metacharacters to mean repetition, existence or ranges in character patterns ● Common metacharacters – Dot (.)  Matches with any character but line feed – Asterisk (*)  Matches 0 or more appearings of the char. or regex preceeding it Master on Free Software

Regular expressions ● Common metacharacters – Brackets []  Match any of the char. classes enclosed by them ●Circumflex (^) after the opening bracket  Inverse match ●Minus ()  Specifies a range ● Metacharacters loose their meaning inside the brackets. Eg: [ at the begining of the list Master on Free Software

Regular expressions ● Common metacharacters – Circumflex (^)  Means line begining – Dollar ($)  Means line ending – Backslash (\\)  Escape next metacharacter ● Example # grep '^.* 15:[09][09].*kernel.*$' /var/log/messages Sep 17 15:04:13 localhost kernel: eth1: remaining active... Sep 17 15:04:18 localhost kernel: eth1: DSPCFG accepted after... Sep 17 15:04:18 localhost kernel: eth1: link up. ... Master on Free Software

Regular expressions ● Metacharacters exclusive to sed, grep, egrep and awk – Slash-brace (/{m,n/})  Matches a range of appearings of the single character immediately before the expression – Variants ● /{m/}  Exactly m repetitions ● /{m,/}  At least m repetitions ● /{m,n/}  Betwee m and n repetitions Master on Free Software

Regular expressions ● Metacharacters exclusive to sed, grep, egrep and awk – Plus (+)  Matches one or more appearings of the character or regex before it – Question mark (?)  0 or 1 appearings – Pipe (|)  Matches the previous expression or the next one – Parenthesis ()  Groups regular expressions Master on Free Software

Regular expressions i man -S7 regex ● Tip: Remember that some characters are interpreted by the shell \$hello\\|hi\$\\ world (hello|hi) world  '(hello|hi) world' ● Regex learning & exploring utilities: – kodos – kregexpeditor – visual-regexp Master on Free Software

Grep ● Grep – Outputs the lines from stdin or file(s) matching a pattern – Only supports basic regexps – Example: Modified files in a local repository svnstatus | grep v nbproject | grep ^M Inverse matching Master on Free Software

Grep ● Egrep – Supports extended syntax ● Rgrep – Supports recursive search – Use -E for extended syntax Master on Free Software

Sed ● Sed: Stream editor for filtering and transforming text sed e script [file] sed f scriptfile [file] ● Substitution s/source/destination/modifiers – Example: for SRC in `find type f` do DEST=`echo $SRC | sed e 's/\\.txt/\\.bak/'` mv $SRC $DEST done Master on Free Software

Sed ● Substitution – Useful modifiers ● g  Global (multiple) replacement per line sed e 's/\\./,/g' ● w file  Write pattern to file sed e 's/kernel/&/w kernel.txt' e 's/gconf/&/w gconf.txt' < /var/log/syslog ● p / !p  Print/hide substitued lines (use n to hide default output). ! inverts the restriction ls lisa | sed n e 's/total/total/p' Master on Free Software

Sed ● Substitution – Using parts of the result ● Whole result (&) ● Substrings: \$, \$, \\1, \\2... sed e 's/\$[09]*\$\\.\$[09]*\$/Integer: \\1, decimal: \\2/' 3452345.46657 Integer part: 3452345, decimal part: 46657 Master on Free Software

Sed ● Patterns and ranges to apply commands – Line: sed n e '12 s/src/dest' – Range of lines: sed n e '12,25 s/src/dest' sed n e '1,25 p' sed n e '12,$ s/src/dest/!p' Master on Free Software

Sed ● Patterns and ranges to apply commands – Matching a pattern: sed n e '/pattern/ s/src/dest' – Matching a range between pattern occurences: sed n e '/begin/,/end/ p' Master on Free Software

Sed ● Command grouping { } sed n e ' /begin/,/end/ { When the pattern is s/1/A/gp matched, the block is executed. s/2/B/gp When not matched, the general command is s/3/C/gp executed. } p ' Master on Free Software

Sed ● More advanced commands – a  Append lines – i  Insert lines – c  Change lines – =  Print line number i man sed http://www.grymoire.com/Unix/Sed.html http://www.panix.com/~elflord/unix/sed.html Master on Free Software

Awk ● Awk: pattern scanning and processing language awk program [file] awk f programfile [file] ● Most common syntax – Line matching awk /regex/ file Master on Free Software

Awk ● Most common syntax i-th word (like if it was a parameter) in the matched – Matching and processing line. $0 is the whole line. awk ' BEGIN {print “Initialization”} /regex1/ {print $1, $2} /regex2/ {print $3, $4} END {print “Final action”} ' file – Master on Free Software

Awk ● Most common syntax – Programming structures inside code blocks { if ($1==”1”) print “Winner”; else print “Looser”; } { x=1; while (x<100) { x=x*2; } } { for (i=0;i<10;i++) { print i; } Master on Free Software

Awk ● Most common syntax – Predefined variables (most important) ● NR  Current count of number of input lines ● NF  Count of number of words in current line ● FILENAME  Current processed file name i man awk http://www.vectorsite.net/tsawk.html Master on Free Software

System integration in free software

0 comments

Notes on slide 1

Favorites, Groups & Events