This document discusses Linux text stream filters and provides examples of common Unix commands used to process and modify text streams. These commands include cat, head, tail, cut, and split. Cat prints the contents of files, head prints the first few lines, tail prints the last few lines, cut extracts parts of each line, and split divides files into smaller parts. The document also covers input/output redirection and how it can be used with filters to modify command output and send it to files.
This lecture discusses a group of techniques to use commands output/Input to feed into other commands or into files. It also covers argument expansion and quoting
Check the other Lectures and courses in
http://Linux4EnbeddedSystems.com
or Follow our Facebook Group at
- Facebook: @LinuxforEmbeddedSystems
Lecturer Profile:
- https://www.linkedin.com/in/ahmedelarabawy
Devops for beginners is basically for the developers who really want to jumps towards DevOps processes and tools. Guide will give basic about what is cloud and virtualisation ?
What are different clouds solutions available?
Most useful and used linux commands with detail description about them.
Linux directory structure and usage.
Basically it will cover all the basics of Linux Administration and cloud concepts.
Healthcare fraud is costing the United States tens of billions of dollars a year and according to William Rudman of AHIMA foundation, the most frightening fact is that the major chunk of fraud happens under the radar and majority of the frauds are left unnoticed; besides that, those of frauds which are identified are not brought into litigation for many years together. Financial fraud and false claims are the most common types of healthcare fraud, according to the AHIMA Foundation report, this includes false claims for medically unnecessary services; false claims that include purposeful overstatement of the amount, number, type, or complexity of the service provided; or false claims that include services that were never rendered or were not rendered on the individuals claimed or by the provider claimed. Another issue associated with fraud and abuse are when physicians refer patients out of financial interest rather than curing the patients. This short presentation is intended to give an overview on two major statutes that help to fight against a variety of fraud, The False Claim Act & Stark Law.
This lecture discusses a group of techniques to use commands output/Input to feed into other commands or into files. It also covers argument expansion and quoting
Check the other Lectures and courses in
http://Linux4EnbeddedSystems.com
or Follow our Facebook Group at
- Facebook: @LinuxforEmbeddedSystems
Lecturer Profile:
- https://www.linkedin.com/in/ahmedelarabawy
Devops for beginners is basically for the developers who really want to jumps towards DevOps processes and tools. Guide will give basic about what is cloud and virtualisation ?
What are different clouds solutions available?
Most useful and used linux commands with detail description about them.
Linux directory structure and usage.
Basically it will cover all the basics of Linux Administration and cloud concepts.
Healthcare fraud is costing the United States tens of billions of dollars a year and according to William Rudman of AHIMA foundation, the most frightening fact is that the major chunk of fraud happens under the radar and majority of the frauds are left unnoticed; besides that, those of frauds which are identified are not brought into litigation for many years together. Financial fraud and false claims are the most common types of healthcare fraud, according to the AHIMA Foundation report, this includes false claims for medically unnecessary services; false claims that include purposeful overstatement of the amount, number, type, or complexity of the service provided; or false claims that include services that were never rendered or were not rendered on the individuals claimed or by the provider claimed. Another issue associated with fraud and abuse are when physicians refer patients out of financial interest rather than curing the patients. This short presentation is intended to give an overview on two major statutes that help to fight against a variety of fraud, The False Claim Act & Stark Law.
On December 5, 2013, Ron Steinkamp, principal, government advisory services at Brown Smith Wallace, presented at the 2013 MIS Training Institute Governance, Risk & Compliance Conference. Ron focused on the following keys to fraud prevention, detection and reporting:
1. Anti-fraud culture
2. Fraud policy
3. Fraud awareness/training
4. Hotline
5. Assess fraud risks
6. Review/investigation
7. Improved controls
A brief talk on systems performance for the July 2013 meetup "A Midsummer Night's System", video: http://www.youtube.com/watch?v=P3SGzykDE4Q. This summarizes how systems performance has changed from the 1990's to today. This was the reason for writing a new book on systems performance, to provide a reference that is up to date, covering new tools, technologies, and methodologies.
You have a system with an advanced programmatic tracer: do you know what to do with it? Brendan has used numerous tracers in production environments, and has published hundreds of tracing-based tools. In this talk he will share tips and know-how for creating CLI tracing tools and GUI visualizations, to solve real problems effectively. Programmatic tracing is an amazing superpower, and this talk will show you how to wield it!
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
4. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Rundown of commands for Process text steams using filters:
commands Overview
4
• cat – concatenate files (or just show a single file without alteration)
• cut – cut chosen text out of each line of a file and display it.
• expand – expand tabs into spaces
• fmt – reformat a text file with a consistent right margin
• head – show the first few (10) lines of a file
• join – join lines of two files on a common field
• nl – print the file with numbered lines
• od – octal dump of a file (or hexadecimal).
• paste – print a number of files side by side
• pr – format for printing (split into pages or columns and add headers)
• sed – stream editor (search and replace, append, cut, delete and more)
• sort – sort in alphabetical order (and numerical order too)
• split – split a single input into multiple files
• tac – print the lines of a file from back to front (backwards cat)
• tail – print the last few lines of a file
• tr – character translation (e.g. upper case to lower case).
• unexpand – convert spaces to tabs (unlike expand).
• uniq – remove duplicate lines from a sorted file
• wc – word count (and line count, and byte count)
5. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Bash makes it possible to redirect the input and output of a command.
•Input - comes from the keyboard (ended by pressing Ctrl+D),
•Output and any errors - are displayed on the screen.
Redirection can change the input of a process, its output and the destination of the errors.
Input and output redirection
5
Redirection Effect of redirection
cmd < file Command reads input from a file
cmd > file Output of command goes to file
cmd 2> file Errors from the command go to a file
cmd >> file Output of a command is added to a file
cmd > file 2>&1 Output and Errors go to a file
cmd >& file
cmd &> file
cmd1 | cmd2 Output from command1 is input for command2
9. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
head – print out the first lines
Selecting parts of a file (Filters that print out various parts of the input they receive)
9
foo:~ # head /var/log/boot.log
Apr 7 08:28:22 foo allrc: syslogd startup succeeded
Apr 7 08:28:22 foo allrc: klogd startup succeeded
Apr 7 08:28:23 foo allrc: portmap startup succeeded
Apr 7 08:27:56 foo rc.sysinit: Mounting proc filesystem: succeeded
Apr 7 08:27:56 foo rc.sysinit: Unmounting initrd: succeeded
Apr 7 08:27:56 foo sysctl: net.ipv4.ip_forward = 0
Apr 7 08:27:56 foo sysctl: net.ipv4.conf.default.rp_filter = 1
Apr 7 08:27:56 foo sysctl: kernel.sysrq = 0
Apr 7 08:28:26 foo lpd: execvp: No such file or directory
Apr 7 08:27:56 foo sysctl: kernel.core_uses_pid = 1
Ex: By default head prints out the first 10 lines of a file.
10. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Selecting parts of a file (Filters that print out various parts of the input they receive)
10
foo:~ $ ls -l / | head -n 6
total 232
drwxr-xr-x 2 root root 4096 Feb 21 15:49 bin
drwxr-xr-x 3 root root 4096 Jan 7 10:25 boot
drwxr-xr-x 5 root root 20480 Jan 10 11:35 data
drwxr-xr-x 21 root root 118784 Apr 7 08:28 dev
drwxr-xr-x 64 root root 8192 Apr 7 08:28 etc
Ex: head can print out a specific number of lines from a file or a stream.
use head to extract an exact number of bytes from an input stream (rather than lines).
Here's how to get a copy of the partition sector of a disk (be careful with that redirection).
foo:~ # head -c 512 < /dev/hda > mbr
foo:~ # ls -la mbr
-rw-r--r-- 1 root root 512 Apr 7 10:27 mbr
11. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
tail – show the end of a file
Selecting parts of a file (Filters that print out various parts of the input they receive)
11
root@foo:root # tail /var/log/messages
Apr 7 11:19:34 foo dhcpd: Wrote 9 leases to leases file.
Apr 7 11:19:34 foo dhcpd: DHCPREQUEST for 10.0.0.169 from
00:80:ad:02:65:7c via eth0
Apr 7 11:19:35 foo dhcpd: DHCPACK on 10.0.0.169 to
00:80:ad:02:65:7c via eth0
Apr 7 11:20:01 foo kdm[1151]: Cannot convert Internet address
10.0.0.168 to host name
Apr 7 11:26:46 foo ipop3d[22026]: connect from 10.0.0.10
(10.0.0.10)
Apr 7 11:26:55 foo ipop3d[22028]: connect from 10.0.0.10
(10.0.0.10)
Apr 7 11:26:58 foo ipop3d[22035]: connect from 10.0.0.3 (10.0.0.3)
Apr 7 11:27:01 foo ipop3d[22036]: connect from 10.0.0.3 (10.0.0.3)
Apr 7 11:29:31 foo kdm[21954]: pam_unix2: session started for user
joe, service xdm
Apr 7 11:32:41 foo sshd[22316]: Accepted publickey for root from
10.0.0.143 port 1250 ssh2
Ex: tail is just like head, but it shows the tail end of the file.
tail can be used to watch a file as it grows.
Run the command tail –f /var/log/messages on one console and then log in on another virtual console.
tail –n 20 file or tail -20 file will show last 20 lines of file. tail –c 20 file will show last 20 characters of a file.
12. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
cut – pull out columns
Selecting parts of a file (Filters that print out various parts of the input they receive)
12
root@foo:root # tail /var/log/messages
Apr 7 11:19:34 foo dhcpd: Wrote 9 leases to leases file.
Apr 7 11:19:34 foo dhcpd: DHCPREQUEST for 10.0.0.169 from
00:80:ad:02:65:7c via eth0
Apr 7 11:19:35 foo dhcpd: DHCPACK on 10.0.0.169 to
00:80:ad:02:65:7c via eth0
Apr 7 11:20:01 foo kdm[1151]: Cannot convert Internet address
10.0.0.168 to host name
Apr 7 11:26:46 foo ipop3d[22026]: connect from 10.0.0.10
(10.0.0.10)
Apr 7 11:26:55 foo ipop3d[22028]: connect from 10.0.0.10
(10.0.0.10)
Apr 7 11:26:58 foo ipop3d[22035]: connect from 10.0.0.3 (10.0.0.3)
Apr 7 11:27:01 foo ipop3d[22036]: connect from 10.0.0.3 (10.0.0.3)
Apr 7 11:29:31 foo kdm[21954]: pam_unix2: session started for user
joe, service xdm
Apr 7 11:32:41 foo sshd[22316]: Accepted publickey for root from
10.0.0.143 port 1250 ssh2
Ex:
Cut can be used to select certain columns of the input stream.
Columns can be defined by either their position, or by being separated by field separators.
15. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Split can split files into more manageable parts, e.g. for FTP uploads. The parts can be recombined with cat.
Selecting parts of a file (Filters that print out various parts of the input they receive)
15
foo:~/download $ ls -la anomy-sanitizer-1.56.tar.gz
-rw-rw-r-- 1 georgem georgem 124356 Oct 22 18:37 anomysanitizer-1.56.tar.gz
foo:~/download $ split -b 32k anomy-sanitizer-1.56.tar.gz
foo:~/download $ ls -la x*
-rw-rw-r-- 1 georgem georgem 32768 Apr 7 11:48 xaa
-rw-rw-r-- 1 georgem georgem 32768 Apr 7 11:48 xab
-rw-rw-r-- 1 georgem georgem 32768 Apr 7 11:48 xac
-rw-rw-r-- 1 georgem georgem 26052 Apr 7 11:48 xad
Ex:
Here's how to use cat to recombine parts (using md5sum to check whether the whole is equal to the sum of
the parts).
foo:~/download $ split -b 32k anomy-sanitizer-1.56.tar.gz part-
foo:~/download $ ls -la part-*
-rw-rw-r-- 1 georgem georgem 32768 Apr 7 11:49 part-aa
-rw-rw-r-- 1 georgem georgem 32768 Apr 7 11:49 part-ab
-rw-rw-r-- 1 georgem georgem 32768 Apr 7 11:49 part-ac
-rw-rw-r-- 1 georgem georgem 26052 Apr 7 11:49 part-ad
foo:~/download $ cat part-* > newfile
foo:~/download $ md5sum newfile anomy-sanitizer-1.56.tar.gz
1a977bad964b0ede863272114bfc2482 newfile
1a977bad964b0ede863272114bfc2482 anomy-sanitizer-1.56.tar.gz
18. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Sorting
18
uniq – discard duplicate lines - uniq is usually used with sort to discard duplicates
foo:~ $ cat /etc/passwd | cut -d : -f 4 | sort -n | fmt
0 0 0 0 0 0 1 2 4 7 12 13 14 25 26 28 29 30 32 37 38 42 43 47 48 50 51
69 74 77 80 89 99 100 500 501 503 504 505 506 507 509 511 512 65534
Ex: Ex: cutting fourth field out of the password file (the group ID) and sorting in numerical order.
fmt is used to make the results display on a single line.
Ex: Ex: same command pipeline, but are removing duplicates with uniq before formatting.
foo:~ $ cat /etc/passwd | cut -d : -f 4 | sort -n | uniq | fmt
0 1 2 4 7 12 13 14 25 26 28 29 30 32 37 38 42 43 47 48 50 51 69 74 77
80 89 99 100 500 501 503 504 505 506 507 509 511 512 65534
19. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Manipulation
19
tr – character set translation - usually used for converting upper case to lower case.
Can also do other character translations. tr -d can remove specific characters from a stream.
foo:~ $ man -P cat man | tr A-Z a-z | less
Ex: Ex: Translating from UPPER CASE to lower case. Asking man to use cat as its pager, instead of
using less, and opening up the man page for man itself.
Translating from lower case to UPPER CASE:
foo:~ $ man -P cat man | tr a-z A-Z | less
Convert file names to lowercase.
foo:/windows/C $ for FILE in * ; do
mv "$FILE" $( echo "$FILE" | tr A-Z a-z ) ; done
Using tr -d to delete the Carriage Returns (r) from a file created with Windows Notepad.
foo:/windows/C $ tr -d 'r' notepad.dos.txt > notepad.unix.txt
Ex:
Ex:
Ex:
21. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Manipulation
21
Joining /etc/passwd and /etc/shadow based on their first field (user name).
Since /etc/passwd and /etc/shadow use a colon to separate fields, it is necessary to use -t: option.
foo:~ # join -t : /etc/passwd /etc/shadow | head
root:x:0:0:root:/root:/bin/bash:$1$LHNUbu7U$oiuhqwd1oiuhqhAdiuHvA0:1
2146:0:99999:7:::
bin:x:1:1:bin:/bin:/sbin/nologin:*:11974:0:99999:7:::
daemon:x:2:2:daemon:/sbin:/sbin/nologin:*:11974:0:99999:7:::
adm:x:3:4:adm:/var/adm:/sbin/nologin:*:11974:0:99999:7:::
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin:*:11974:0:99999:7:::
sync:x:5:0:sync:/sbin:/bin/sync:*:11974:0:99999:7:::
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown:*:11974:0:99999:7:::
halt:x:7:0:halt:/sbin:/sbin/halt:*:11974:0:99999:7:::
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin:*:11974:0:99999:7:::
Ex:
join allows to specify which particular field to join on, and also which particular fields should appear in the
output (similar to cut)
23. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Manipulation
23
sed – stream editor - sed does text transformations on an input stream.
foo:~ $ echo "Hi Fred, how are you" | sed 's/Fred/Joe/'
Hi Joe, how are you
foo:~ $ echo "Hi Fred, how is Fred?" | sed 's/Fred/Joe/'
Hi Joe, how is Fred?
foo:~ $ echo "Hi Fred, how is Fred?" | sed 's/Fred/Joe/g‘
Hi Joe, how is Joe?
Ex:
•sed works by making only one pass over the inputs.
•A sed program consists of one or more sed commands which are applied to each line of the input.
•A command may be prefixed by an address range indicating lines for which the command performes.
sed commands: s/PATTERN/REPLACEMENT/g
search and replace. If you add g at end - search and replace applies as many times as possible to single line.
You can also use i at the end of the s command to make the search case insensitive
24. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Manipulation
24
foo:~ $ ls /bin/ | fmt -30 | nl | sed '4,15d'
1 arch ash ash.static awk
2 basename bash cat chgrp
3 chmod chown chvt cp cpio csh
16 true umount uname usleep vi
17 vim zcat zsh
foo:~ $ ls /bin | fmt -40 | nl | sed '/e/ d'
2 bash cat chgrp chmod chown chvt cp
13 vi vim zcat zsh
Ex:
sed commands: d – delete the line.
You need to select the lines as explained in the next paragraph.
The commands are most useful if you specify a range of lines to which the command applies.
Here's how to specify specific lines for the d command:
•/PATTERN/d – delete all lines contains the pattern
•4d – delete line 4
•4,10d – delete lines 4 to 10
•6,$d – delete from line 6 to the last line
29. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Manipulation
29
paste – paste two files together
foo:~/tmp $ cat file2
cabbage green leaves
cat white cat-hair
piano brown wood
foo:~/tmp $ cat file1
cat animal
cabbage vegetable
piano mineral
coal mineral
foo:~/tmp $ paste file2 file1 | expand -t 22
cabbage green leaves cat animal
cat white cat-hair cabbage vegetable
piano brown wood piano mineral
coal mineral
Ex:
Using paste is like taking two printouts of two files and sticking the right margin of one to the left margin of
the other. The glue between the files is a tab stop.
You can specify a delimiter between the files which is not a tab space with option -d
30. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Formatting
30
fmt – format nicely - format text into a neatly word wrapped structure. Default right margin is 75
foo:~ $ ls /bin | fmt
arch ash ash.static aumix-minimal awk basename bash bash2 bsh cat
chgrp
chmod chown cp cpio csh cut date dd df dmesg dnsdomainname doexec
domainname dumpkeys echo ed egrep env ex false fgrep gawk gettext
grep gtar gunzip gzip hostname igawk ipcalc kbd_mode kill link ln
loadkeys login ls mail mkdir mknod mktemp more mount mt mv netstat
nice
nisdomainname pgawk ping ps pwd red rm rmdir rpm rvi rview sed
setfont
setserial sh sleep sort stty su sync tar tcsh touch true umount
uname
unicode_start unicode_stop unlink usleep vi vi.minimal view
ypdomainname
Ex:
foo:~ $ ls /bin | fmt -40 | head
arch ash ash.static aumix-minimal
awk basename bash bash2 bsh cat chgrp
chmod chown cp cpio csh cut date dd df
dmesg dnsdomainname doexec domainname
dumpkeys echo ed egrep env ex false
fgrep gawk gettext grep gtar gunzip
gzip hostname igawk ipcalc kbd_mode
kill link ln loadkeys login ls mail
mkdir mknod mktemp more mount mt mv
netstat nice nisdomainname pgawk ping
32. CoreLinuxforRedHatandFedoralearningunderGNUFreeDocumentationLicense-Copyleft(c)AcácioOliveira2012
Everyoneispermittedtocopyanddistributeverbatimcopiesofthislicensedocument,changingisallowed
Process text streams using filters
Formatting
32
od – octal dump (and other formats)
Ex:
foo:/tmp $ echo "Hello World" > hello
foo:/tmp $ cat hello
Hello World
od, oddly enough, does not just print out in octal, but in other formats.
foo:/tmp $ od hello
0000000 062510 066154 020157 067527 066162 005144
0000014
od's behaviour is rather odd when it is used without any arguments. It prints out the octal
value of two pairs of bytes in the file. Sometimes this is useful, but usually it isnot.
Using -t switch tells od to use a specific format type (the default is od –t o2).
od –t c means character format. You will notice that file ends in a newline character (n).
foo:/tmp $ od -t c hello
0000000 H e l l o W o r l d n
0000014
od –t d1 specifies decimal format, with a width of one. character encoding is ASCII.
Ex:
Ex:
foo:/tmp $ od -t d1 hello
0000000 72 101 108 108 111 32 87 111 114 108 100 10
0000014