sed, known as the “streams editor”, is a powerful tool for text manipulation on UNIX systems. Unlike vim, sed can operate on files of limitless size. The session will cover some of the intermediate concepts such as grouping and back referencing, regular expressions, replacing the nth occurrence of a pattern, operation on every nth line and so on. We will spice up the session with a few “sed one-liners/idioms”
awk is a very versatile programming language for working on text files. It is more powerful than sed but less complex than C. It is an excellent filter and report writer. In this class I will go over the elements and features of gawk, (the Free Software foundation version of awk), examples and a few one-liners.
This course provides you with skills to
* Develop sed and awk scripts
* Use sed and awk to automate common tasks
* Use sed and awk to create formatted reports
Prerequisites
* Basic understanding of UNIX / Linux Operating System
* Knowledge of basic UNIX / Linux commands
Intended Audience
* System Administrators, Testing Professionals, and Software Developers working in the UNIX / Linux environment
Oracle database supports perl- and POSIX-compatible regular expressions with five elegant and powerful functions: REGEXP_REPLACE, REGEXP_SUBSTR, REGEXP_INSTR, REGEXP_LIKE, and REGEXP_COUNT.
This session will demonstrate their nuances and how to use them effectively for data cleansing, manipulation and selection, for validating things such as Social Security Numbers, credit cards, IP addresses, phone numbers, DNAs, XMLs, for extracting things such as email-ids, hostnames from URLs and strings, and for transposing delimited columns to rows. There will be a demo of a few tricky examples taken from forums.oracle.com and asktom.oracle.com.
The session will end with fuzzy matching and optimization techniques, and things to watch out for.
http://docs.oracle.com/cd/E11882_01/appdev.112/e25518/adfns_regexp.htm
awk is a very versatile programming language for working on text files. It is more powerful than sed but less complex than C. It is an excellent filter and report writer. In this class I will go over the elements and features of gawk, (the Free Software foundation version of awk), examples and a few one-liners.
This course provides you with skills to
* Develop sed and awk scripts
* Use sed and awk to automate common tasks
* Use sed and awk to create formatted reports
Prerequisites
* Basic understanding of UNIX / Linux Operating System
* Knowledge of basic UNIX / Linux commands
Intended Audience
* System Administrators, Testing Professionals, and Software Developers working in the UNIX / Linux environment
Oracle database supports perl- and POSIX-compatible regular expressions with five elegant and powerful functions: REGEXP_REPLACE, REGEXP_SUBSTR, REGEXP_INSTR, REGEXP_LIKE, and REGEXP_COUNT.
This session will demonstrate their nuances and how to use them effectively for data cleansing, manipulation and selection, for validating things such as Social Security Numbers, credit cards, IP addresses, phone numbers, DNAs, XMLs, for extracting things such as email-ids, hostnames from URLs and strings, and for transposing delimited columns to rows. There will be a demo of a few tricky examples taken from forums.oracle.com and asktom.oracle.com.
The session will end with fuzzy matching and optimization techniques, and things to watch out for.
http://docs.oracle.com/cd/E11882_01/appdev.112/e25518/adfns_regexp.htm
From my November 3, 2011 talk at MNPHP. Regular expressions are a powerful tool available in nearly every programming language or platform, including PHP. I go over the history of POSIX vs. PCRE, examples in PHP, and optimizations on how to write faster expressions.
Regular Expressions in PHP, MySQL by programmerblog.netProgrammer Blog
This PPT explains, how to use regular expressions in PHP. PHP has two types of regular expressions Perl Style and Posix Style.
Read detailed tutorials on http://programmerblog.net
From my November 3, 2011 talk at MNPHP. Regular expressions are a powerful tool available in nearly every programming language or platform, including PHP. I go over the history of POSIX vs. PCRE, examples in PHP, and optimizations on how to write faster expressions.
Regular Expressions in PHP, MySQL by programmerblog.netProgrammer Blog
This PPT explains, how to use regular expressions in PHP. PHP has two types of regular expressions Perl Style and Posix Style.
Read detailed tutorials on http://programmerblog.net
vim is the "improved" version of the popular "vi" editor in UNIX/Linux environments.
With this powerful editor, do you know how to:
delete lines between two different search strings, repetitively?
replace multiple spaces with a single space OR trim all the leading or trailing spaces?
convert to upper/lower case, title/sentence case, to toggle case?
edit many files with multiple windows or tabs, and execute the same command on all at the same time?
delete blank lines or replace multiple blanks lines with one line?
search and replace with back referencing, and regular expressions?
identify palindromes, repeating words?
format lines (center-, left-, right-align) like you do in MS Word, but only lines that meet certain conditions?
go back to previous version as of N versions/seconds/minutes/hours ago even after saving with :w?
In this three-hour class, you will learn how to use vim effectively with shortcuts, markers, regular expressions, substitutions, grouping and back referencing, mapping, non-volatile buffers, recalling the nth delete, recording and replaying with macros, batching, and customizations. Concepts will be explained with demos. You will come away knowing how to simplify mundane editing tasks and become more productive.
Recognizing patterns in a sequence of rows has been a capability that was widely desired, but not possible with SQL until now. There were many workarounds, but these were difficult to write, hard to understand, and inefficient to execute. Beginning in Oracle Database 12c, you can use the MATCH_RECOGNIZE clause to achieve this capability in native SQL that executes efficiently. This presentation discusses how to do this.
Are you an Oracle developer or a DBA?
Do you know the difference between aggregate and analytic functions?
Without complex sub-queries or self-joins, do you know how to:
Calculate running/cumulative totals and moving/centered averages?
List products with revenues above or below their peers or product groups?
Compute the ratio of one category’s sales to the total sales?
Select the Top-N or Top N % of the customers/products?
Classify advertisers into quartiles/n-tiles based on the revenue potential?
Compare period-over-period (year-over-year, month-over-month) growth and rank advancement?
Convert rows into columns (pivot), columns into rows (unpivot) or aggregate strings?
Perform what-if analysis and hypothetical ranking?
Analytic functions are more performant because tables need to be scanned only once. They make you more productive because there is no need to write procedural code. No wonder Tom Kyte, a well-respected Oracle guru, says analytic functions are the best thing to happen after the sliced bread.
In the first half, I will cover the basics of the various analytic functions:
Ranking: RANK, DENSE_RANK, ROW_NUMBER, NTILE, CUME_DIST, PERCENTILE_RANK
Windowing: SUM, AVG, MAX, MIN, FIRST_VALUE, LAST_VALUE
Reporting: RATIO_TO_REPORT
Others: FIRST/LAST, LEAD/LAG, hypothetical ranking,
In the second half, I will show how powerful these functions are with a few examples.
If there is time, I will cover enhanced aggregation (ROLLUP, CUBE, GROUPING SET extensions to GROUP BY clause)
This class would be useful for both developers and DBAs alike, especially for those working in Analytic, Business Intelligence, and Datawarehouse environments.
Are you already an expert in analytic functions? Then come and help me refine the content.
For more info, read
http://download.oracle.com/docs/cd/E11882_01/server.112/e16579/analysis.htm
http://download.oracle.com/docs/cd/E11882_01/server.112/e16579/aggreg.htm
rollup, cross-tabulation across different dimensions using ROLLUP, CUBE and GROUPING SETS extension to GROUP BY clause
, most active time-periods (i.e. days when the most number of tickets are open in BZ, hours with the most take-off and landings, months with the highest sales, 5-minute periods with the maximum number of calls made, etc)
data densification?
their rank last year, this year, rank growth, running/cumulative total (Year-To-Date/Month-To-Date summation), moving averages, Year-Over-Year comparison, sales projection, average/min/max time between one sale and the next sale, products with above and below average sales.
overall average, sum, departmental average, sum, ranking, job wise ranking in one SQL.
University of North Texas 2 The sed Stream Editor .docxouldparis
University of North Texas
2
The sed Stream Editor
3
The sed Stream Editor
• sed is a non-interactive, line-oriented stream editor that
processes one line at a time
– Useful in text processing and especially performing in-place
substitution
– sed can make global substitutions of matched regex patterns with
specific text
• Example
– How change all occurrences of word "the" or "The" to uppercase
"THE" in file called file1?
sed -r "s/(The|the)/THE/g" file1
4
The sed Stream Editor
• Usage:
sed -r "s/REGEX/TEXT/g" filename
– Substitutes (replaces) occurrence(s) of REGEX with the given TEXT
– If filename is omitted, reads from standard input
– sed has other uses, but most can be emulated with substitutions
– Resulting output to terminal
• If wanted permanent changes, need redirect output to new file or
make changes in-place using –i option
• Example
– Replaces all occurrences of 143 with 390 in file2.txt
sed -r "s/143/390/g" file2.txt
5
The sed Stream Editor
• sed is line-oriented; processes input a line at a time
-r option makes regexes work better
– Recognizes ( ) , [ ] , * , + the right way, etc.
s for substitute
g flag after last / asks for a global match (replace all)
• Special characters must be escaped to match them literally
sed -r "s/http:\/\//https:\/\//g" urls.txt
• sed can use delimiters besides / to make more readable
sed -r "s#http://#https://#g" urls.txt
• Example
sed -r "s/([A-Za-z]+), ([A-Za-z]+)/\2 \1/g" names.txt
6
sed Usage
• Edit files too large for interactive editing
• Edit any size files where editing sequence is too complicated to
type in interactive mode
• Perform “multiple global” editing functions efficiently in one
pass through the input
• Edit multiples files automatically
• Good tool for writing conversion programs
7
The sed Command
or
'command'
8
sed Syntax
sed [-n] [-e] ['command'] [file…]
sed [-n] [-f script] [file…]
• Options
–n only print lines specified with print command (or 'p'
flag of substitute ('s') command)
–f script next argument is filename containing editing
commands
If first line of script is "#n", acts as if -n had been
specified
–e command next argument is an editing command rather than
filename, useful if multiple commands are specified
9
How Does sed Work?
• sed reads line of input
– Line of input is copied into a temporary buffer called pattern space
– Editing commands are applied
• Subsequent commands are applied to line in the pattern space,
not the original input line
• Once finished, line is sent to output (unless –n option was used)
– Line is removed from pattern space
• sed reads next line of input, until end of file
• Note that input file is unchanged!
10
sed Scripts
• A script is nothing more than a file of commands
• Each command consists of an address and an action, where
the address can be a regular expression or line numbe ...
When performing an investigation it is helpful to be reminded of the powerful options available to the investigator. This document is aimed to be a reference to the tools that could be used.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
3. What is sed
3
non-interactive
non-screen oriented
Line oriented
Input file can be any size
Input file not affected
4. sed syntax
4
sed [options] 'cmd' in_file(s)
sed [options] 'cmd' in_file(s) [> out_file]
Standard input: sed [options] 'cmd' < in_file [>
out_file]
Pipelined input: command | sed [options] 'cmd' [>
out_file]
5. Simple examples
5
Example Explanation
sed 's/pat1/pat2/g' file Substitute all occurrences of pat1 with pat2
sed '/pat1/d' file delete lines containing pat1 from file
sed '/pat1/w newfile' file save lines containing pat1 to newfile
sed –n '30,40p' file Print lines 30 to 40
sed '10q' file Print the top 10 lines
sed –e 's/pat1/pat2/' –e Substitute pat1 with pat2, and pat3 with pat4
's/pat3/pat4/' file
sed Substitute pat1 with pat2, and pat3 with pat4
's/pat1/pat2/;s/pat3/pat4
/' file
6. Different sed options/switches
6
Range Remarks
-e Used when multiple commands are used on the
command line
-n Suppress automatic printing of pattern space
-f script-file The commands in the script-file get executed
-r Use extended regular expressions in the script. With
this option, characters such as (, ), {, }, | become meta
characters, and don't have to be escaped.
-s Consider files as separate rather than as a single
continuous long stream
-i in-place editing of the input file
7. The "-f" option - example
7
cat mysed.txt
/pat1/ s/this/that/g
/pat2/ s/before/after/
sed –f mysed.txt in_file
apostrophes not used
8. With and without "-s" option -
Comparison
8
Lines Lines sed –n '1,10p' f1, f2 sed –ns '1,10p' f1, f2
in f1 in f2
4 5 9 lines 4 lines from f1, 5 lines from f2
6 6 6 lines from f1, 4 lines from f2 6 lines from f1, 6 lines from f2
12 10 10 lines from f1. No lines from 10 lines from f1, 10 lines from
f2 f2
9. Address Specification
9
Range Remarks
10 Just the line 10
1,10 Lines between 1 and 10
10,$ Line 10 to end of file
10,+3 Line 10 and 3 lines below (lines 10, 11, 12 and 13)
10~3 every third line after line 10
10, ~3 Line 10 and the next multiple of 3 (lines 10,11 and 12)
/pat1/ All lines containing pat1
/pat1/,+3 Lines containing pat1 and three lines following it
/pat1/,~3 Lines containing pat1 and up to the multiple of 3
/pat1/, 20 Lines between the line containing pat1 and line 20 if pat1 appears
before line 20. Otherwise, just the line containing pat1
/pat1/, /pat2/ lines between the line containing pat1 and line
containing pat2
10. Address Specification with negation
10
Range Remarks
10! All lines except 10 (! is the negation indicator)
1,10! Lines from 11 to end of the file
10,$! Lines 1 to 9
10,+3! All lines except 10, 11, 12 and 13
10~3! All lines except line 10 and every third line after that
10, ~3! All lines except lines 10, 11, and 12
/pat1/! All lines not containing pat1
/pat1/,+3! All line except lines containing pat1 and three lines
following it
/pat1/,~3! All lines containing pat1 and up to the multiple of 3
11. Regular Expressions
11
Meta character Meaning
. Matches any single character except newline
* Matches zero or more of the character preceding it
e.g.: bugs*, table.*
^ Denotes the beginning of the line. ^A denotes lines starting
with A
$ Denotes the end of the line. :$ denotes lines ending with :
Escape character (., *, [, , etc)
[] matches one or more characters within the brackets. e.g.
[aeiou], [a-z], [a-zA-Z], [0-9], [:alpha:], [a-z?,!]
[^] matches any characters others than the ones inside brackets.
eg. ^[^13579] denotes all lines not starting with odd numbers,
[^02468]$ denotes all lines not ending with even numbers
<, > Matches characters at the beginning or end of words
12. Extended Regular Expressions
12
Meta character Meaning
| alternation. e.g.: ho(use|me), the(y|m), (they|them)
+ one or more occurrences of previous character. a+ is same as
aa*)
? zero or one occurrences of previous character.
{n} exactly n repetitions of the previous char or group
{n,} n or more repetitions of the previous char or group
{,m} zero to m repetitions on the previous char or group
{n, m} n to m repetitions of previous char or group
(....) Used for grouping
sed –r ... the "-r" option may have to be used on some version of sed for
extended regular expressions to work
e.g.: sed –r '/(pat1|pat2) s/pat3/pat4/'
sed –rn '/(pat1|pat2){3,}/ p'
13. Regular Expressions – Examples
13
Example Meaning
.{10,} 10 or more characters. Curly braces have to
escaped
[0-9]{3}-[0-9]{2}-[0-9]{4} Social Security number
([1-9]{3})[1-9]{3}-[0-9]{4} Phone number (xxx)yyy-zzzz
[0-9]{2,3}.[0-9]{1,3}.[0- IP address format
9]{1,3}.[0-9]{1,3}
[0-9]{3}[ ]*[0-9]{3} Postal code in India
[0-9]{5}(-[0-9]{4})? US ZIP Code + 4
14. Substitution – Format and Options
14
[address[!]]s/pat1/pat2/[options]
Options: g – global, w – write, i – ignore case, p –
print, n – nth occurrence
: or ; +, @ or any other character including space
could also be used as the delimiter. Useful when / is
part of the search or replacement string.
15. Substitution - Examples
15
Example Explanation
sed 's/pat1/pat2/' fn Substitute the FIRST occurrence of pat1 with pat2
sed 's/pat1/pat2/g' fn Substitute ALL occurrences of pat1 with pat2
sed 's/pat1/pat2/3' fn Substitute the third occurrence of pat1 with pat2
sed 's/pat1/pat2/3g' fn Substitute all but the first two occurrences of pat1
with pat2
sed 's/pat1/pat2/gi' fn Substitute ALL occurrences of pat1 with pat2
ignoring the case
sed 's/pat1/pat2/giw Write to new_files lines containing pat1
new_file' fn substituting with pat2
sed –n 's/pat1/pat2/gp' fn Print lines containing pat1 substituting with pat2
16. Substitution – Examples contd
16
Example Explanation
sed '/pat1/ s/pat2/pat3/g' fn Substitute all occurrences of pat2 with pat3 on
lines containing pat1
sed '/pat1/! s/pat2/pat3/g' Substitute all occurrences of pat2 with pat3 on
fn lines NOT containing pat1
sed '/pat1/,/pat2/ Substitute all occurrences of pat3 with pat4 on
s/pat3/pat4/g' fn lines between pat1 and pat2 (inclusive)
sed '1,100 s/pat1/pat2/gi' fn Substitute ALL occurrences of pat1 with pat2
ignoring the case between lines 1 and 100
sed '/pat1/,/pat2/! Substitute all occurrences of pat3 with pat4 on
s/pat3/pat4/g' fn lines NOT between pat1 and pat2 (inclusive)
sed '1,100! s/pat1/pat2/3i' fn Substitute the third occurrence of pat1 with pat2
ignoring the case from 101 to end of file
sed '2~5 s/pat1/pat2/w Write every 5th line starting with the 2nd line,
new_file ' fn substituting pat1 with pat2
17. Substitution – Examples contd.
17
Example Explanation
sed Substitute all occurrences of either pat1 or pat2
's/(pat1|pat2)/pat3/g' fn with pat3
sed –r 's/(pat1|pat2)/pat3/g' Same as above. With the –r option, parenthesis and
fn alternation characters don't have to be escaped
sed –r Substitute all occurrences of either abcpat1 or
's/abc(pat1|pat2)/pat3/g' fn abcpat2 with pat3
sed 's/<pat1/pat3/g' fn Substitute all occurrences pat1 that begin a word
with pat3.
sed 's/pat1>/pat3/g' fn Substitute all occurrences pat1 that is end of a
word with pat3
sed 's/<pat1>/pat3/g' fn Substitute all occurrences of the word pat1 with pat3.
Note: The angular brackets (< and >) have to be escaped
even with the –r option.
sed 's/[a-z]/u&/g' fn Substitute all lower case letters to upper case letters
sed 's/./&&/g' fn Double each character
18. Grouping and Back Referencing
18
Parts of strings in the Search/Left-hand side can be
grouped and referenced in the Replacement/Right-
hand side
Up to nine groups possible (1, 2, ..9)
Groups can be nested or referenced back on the
Search side
Same group can be referenced any number of times
19. Substitution with Grouping and Back
Referencing. Examples
19
Command Explanation
sed 's/^(.*):(.*)/2:1/' fn Swap two fields delimited with :. "column
A:column B" becomes "column B:column A"
sed –r 's/^(.*):(.*)/2:1/' fn Same as above. With the –r option, the
parentheses don't have to be escaped.
sed –r 's/^([^:]*):([^:]*)/2:1/' Same as above. Exchanges only the first two
fn columns even if there are more.
sed –r 's/^(This (.*) nested)/2 Group 1 contains everything between "This ..
1/' fn nested". Group 2 contains just the characters
between "This" and "nested".
20. Substitution with Grouping and Back
Referencing. Examples
20
Command Explanation
sed –r Print lines with six or seven character long
'(w)(w)(w)w?321/p' fn palindromes
sed –r Convert six char palindrome strings to
's/(.)(.)(.)321/123123/g repetitive strings. Note: Any embedded or
' fn trailing six spaces also will match
sed -r ':a;s/(^|[^0-9.])([0- Add comma as thousands separator (Think
9]+)([0-9]{3})/12,3/g;ta' how easy it would have if we were to do this
numbers.txt from the left side)
sed –r 's/(.*)/11/' fn Concatenate the string at the line level
sed –r 's:(.*):11:' fn Same as above, but using : as the delimiter
between search and replacement strings
21. Inserting text with "i" and "a"
21
Example Meaning
sed '4i Inserts two lines before line 4
my text 1
my text 2' in_file
sed '/pat1/ i Inserts two lines before every line that contains
my text 1 pat1
my text 2' in_file
sed '4a Inserts two lines after line 4
my text 1
my text 2' in_file
sed '$a Inserts two lines after the end of the file
my text 1
my text 2' in_file
22. Changing text
22
Example Meaning
sed '4c Replace line 4 with the two new lines
my text 1
my text 2' in_file
sed '/pat1/ c Replace all lines containing pat1 with the two
my text 1 new lines
my text 2' in_file
sed '/pat1/, /pat2/ c Replace all lines between pat1 and pat2 with the
my text 1 two new lines
my text 2' in_file
Change affects the whole line.
Substitute just the matching words or strings on the line
23. Deleting lines
23
Command Result
sed '10d' in_file Delete the 10th line
sed '1,10d' in_file Delete lines between 1 and 10
sed '10,$d' in_file Delete lines from 10 to end of file
sed '10,+3d' in_file Delete line 10 and 3 lines below (lines 10, 11, 12 and 13)
sed '10~3d' in_file Delete every third line after line 10
sed '10, ~3d' in_file Delete line 10 and up to the next multiple of 3 (lines 10,11 and
12)
sed '/pat1/d' in_file Delete all lines containing pat1
sed '/pat1/,+3d' in_file Delete all lines containing pat1 and three lines following it
sed '/pat1/,~3d' in_file Delete lines containing pat1 and up to next the multiple of 3
sed '/pat1/!d' in_file Delete all lines NOT containing pat1
sed –r '/[0-9]{8,}/d' in_file Delete lines containing 8 or more digits
24. Printing lines
24
Command Result
sed –n '10,/pat1/p' in_file Print lines from 10 to next line containing pat1
sed –n '10,+3p' in_file Print line 10 and 3 lines below (lines 10, 11, 12 and 13)
sed –n '10~3p' in_file Print every third line after line 10
sed –n '10, ~3p' in_file Print line 10 and up to the next multiple of 3 (lines 10,11 and
12)
sed –n '/pat1/p' in_file Print all lines containing pat1
sed –n '/pat1/,+3p' in_file Print all lines containing pat1 and three lines following it
sed –n '/pat1/,~3p' in_file Print lines containing pat1 and up to next the multiple of 3
sed –n '/pat1/,/pat2/p' Repetitively print lines between the ranges containing
in_file pat1 and pat2
sed –n '/pat1/!p' in_file Print all lines NOT containing pat1
sed -n '/([0-9]{3})[0- Print lines containing phones numbers.
9]{3}-[0-9]{4}/p'
in_file
25. A note on the –r option
25
Command Result
sed -n '/([0-9]{3})[0- Print lines containing phones numbers like (408)806-
9]{3}-[0-9]{4}/p' 8330, (408)349-3699
in_file
sed -nr '/([0- Same as above with the –r option. Note the escaping of
9]{3})[0-9]{3}-[0- parentheses. Without escaping, ( and ) become meta
9]{4}/p' in_file characters, not part of the search string.
26. Printing with and without "-n"
option - Comparison
26
Command Comments
sed '1,10p' fn Lines 1 to 10 are printed twice. Rest of the lines are
printed once
sed –n '1,10p' fn Lines 1 to 10 are printed just once.
sed 's/pat1/pat2/p' fn Lines containing pat1 are printed twice after
substituting pat1 with pat2. Other lines are printed
once
sed –n 's/pat1/pat2/p' Only lines containing pat1 are printed after
fn substituting pat1 with pat2
27. Inserting/reading from a file
27
Command Result
sed '10r my_file' in_file Insert my_file after the 10th line
sed '1,10r my_file' in_file Insert my_file after each line between 1 and 10
sed '$r my_file' in_file Append my_file at the end
sed '10,+3r my_file' in_file Insert my_file after every line between10 and 13.
sed '10~3r my_file' in_file Insert my_file every third line after line 10
sed '/pat1/,/pat2/r my_file' Insert my_file after every line between pat1 and pat2
in_file
28. Writing selectively
28
Command Result
sed '1,10w nf' in_file Write lines 1 to 10 to new file "nf"
sed '/^pat1/w nf' in_file Write all lines beginning with pat1 to nf
sed '/pat1$/w nf' in_file Write lines that end with pat1
sed '/<pat1>/w nf' in_file Write lines that contain the WORD pat1 to nf
sed –e '1~2w odd_lines' Write odd numbered lines to odd_lines, and even
-e '2~2w even_lines` numbered files to even_lines
in_file
sed '/pat1/, /pat2/w nf' Write all lines between pat1 and pat2 to nf
in_file
sed –r '/[0-9]{3}-[0-9]{2}-[0- Write lines containing Social Security Number nnn-nn-nnnn to the
9]{4}w ssn' in_file file ssn. Note the use of –r option. Otherwise, { and } have to be
escaped.
sed –r '/(.)(.)(.)321/w Write lines containing palindromes to palin.txt
palin.txt' in_file
29. Transliterating (like ‘tr’)
29
Command Result
y/source/dest/ One to one character by character substitution.
source and dest strings have to be of the same
length
sed 'y/13579/aeiou/' fn replace 1 by a, 3 by e, and so on
30. Quitting from sed
30
Command Result
sed '15q' in_file Quit after the 15th line. Print 14 lines
sed '/<[Ee]nd>/q' in_file Quit after encountering the word End or end
31. Grouping multiple commands with { and }
31
sed –n '/pat1/,/pat2/ {
s/pat3/pat4/g
s/pat5/pat6/g
w new_file
p
}' inp_file
32. One liners
32
Command Result
sed 'p' in_file Duplicate all the lines one below the other
sed '/^$/ d' in_file Delete the blank lines from the input file
sed ‘s/./&:/80’ Add a colon after the 80th character. (replace the
eightieth occurrence of the pattern with itself and a
colon)
sed –r ‘s/^(.*) 1/1/' fn Replace duplicate strings
sed G in_file Add a blank line after everyline.
find . –type f –name Replace all occurrences of TableA with TableB in all
my_files.*.sql –exec sed –i my SQL scripts
‘s/TableA/TableB/g’ {} +
More oneliners http://www.catonmat.net/blog/sed-one-liners-
explained-part-one/
33. Additional concepts not covered
33
labels, branching to them
h, H: copy/append patternspace to holdspace
g, G: copy/append holdspace to patternspace
x: exchange the contents of the hold and pattern
spaces
34. References
34
sed & awk by Dale Dougherty & Arnold Robins
http://www.grymoire.com/Unix/Sed.html
http://sed.sourceforge.net/sedfaq3.html
http://en.wikipedia.org/wiki/Sed
http://groups.yahoo.com/group/sed-users/
http://sed.sourceforge.net/sed1line.txt
36. Unanswered questions
36
How to simulate tail with sed?
How to substitute the nth to mth occurrences of pat1
with pat2?
How to substitute the last N occurrences or the nth
occurrence from the end?
How to identify palindrome of any length?
Editor's Notes
In /pat1/,+3 if pat1 again appears within the next three lines, it will be ignored.