中国DOS联盟

Collect some good articles related to sed and post them here. Add and supplement links and resources in this thread.

Repost Note: The author is unknown, and the original link is unknown.
Summary of Sed Commands

1．:
Usage:
:lable
Mark a line in the script for control transfer by b or t. Label can contain up to 7 characters

2．=
Usage:
==
Write the addressed line to standard output

3．a
Usage:
a
text
Append text after each line that matches address. If text has more than one line, the newline before these lines must be "hidden" with a backslash. Text will be ended by the first newline that is not hidden in this way. Text is not available in the pattern space and subsequent commands cannot be applied to it. When the list of edit commands is exhausted, the result of this command will be sent to standard output regardless of what happens to the current line in the pattern space.

4．b
Usage:
]b
Unconditionally transfer control to :label elsewhere in the script. That is, the command following label is the next command applied to the current line. If label is not specified, control will reach the end of the script, so no more commands will act on the current line.

5．c
Usage:
]c
text
Replace (change) the lines selected by the address with text. When a line range is specified, all these lines are replaced as a group by a copy of text. The newline after each text line must be escaped with a backslash, except for the last line. In effect, the contents of the pattern space are deleted, so subsequent commands cannot be applied to it (or to text)

6．d
Usage:
]d
Delete the line from the pattern space. Therefore, the line is not passed to standard output. A new input line is read and edited with the first command of the script.

7．D
Usage:
]D
Delete part of the multi-line pattern space created by command N (up to the embedded newline) and resume editing with the first command of the script. If this command makes the pattern space empty, then a new input line is read, same as if command d were executed.

8．g
Usage:
]g
Copy the contents of the hold space (see commands h or H) into the pattern space and clear the current contents.

9．G
Usage:
]G
Append the contents of the hold space (see commands h or H) after a newline to the pattern space. If the hold space is empty, a newline is added to the pattern space.

10．h
Usage:
]h
Copy the contents of the pattern space to the hold space, i.e., a special temporary buffer. The current contents of the hold space are cleared.

11．H
,address2]]H
Append a newline and the contents of the pattern space to the hold space. This command appends a newline even if the hold space is empty.

12．i
Usage:
i
text
Insert text before each line that matches address

13．l
Usage:
]l
List the contents of the pattern space, representing non-printable characters as ASCII codes. Long lines are wrapped.

14．n
Usage:
]n
Read the next input line into the pattern space. The current line is sent to standard output. The new line becomes the current line and increments the line counter. Control is transferred to the command following n, not back to the top of the script.

15．N
Usage:
]N
Append the next input line to the contents of the pattern space; the newly added line is separated from the current contents of the pattern space by a newline (this command is used to implement cross-line pattern matching. Using n to match an embedded newline allows multi-line pattern matching)

16．p
Usage:
]p
Print the addressed line. Note that this will cause duplicate output unless the default output is limited with the "#n" or "-n" command-line option. Often used before commands that change flow control (d, n, b) and may prevent the current line from being output.

17．P
Usage:
]P
Print the first part of the multi-line pattern space created by command N (the directly embedded newline). If N has not been applied to a line, it is the same as p.

18．q
Usage:
q
Exit when address is encountered. The addressed line is first written to output (if default output is not restricted), including text appended to it by previous a or r commands.

19．r
Usage:
r file
Read the contents of file and append it to the contents of the pattern space. A space must be kept between r and the filename file.

20．s
Usage:
]s/pattern/replacement/
Replace each occurrence of pattern in the addressed lines with replacement. If a pattern address is used, then pattern // represents the last specified pattern address. The following flags can be specified:
n Replace the nth /pattern/ in each addressed line. N is any number between 1 and 512, and the default is 1.
g Replace all /pattern/ in each addressed line, not just the first one
p Print the line if the replacement is successful. If multiple replacements are successful, multiple copies of this line will be printed.
w file Write the line to file if one replacement occurs. Up to 10 different files can be opened.
Replacement is a string used to replace the content matching the regular expression. Only the following characters have special meaning in the replacement part:
& Replace with the content matched by the regular expression
n Match the nth substring (n is a number), which was previously specified in the pattern with "(" and ")"
When including the "&" symbol, backslash (\), and the delimiter of the replacement command in the replacement part, they can be escaped. Additionally, it is used to escape newlines and create multi-line replacement strings.
Number flags
s/pattern/replacement/flag
If flag is a number, then the match at a certain position on a line is specified. If there is no number flag, the replacement command only replaces the first match, so "1" can be considered the default number flag.
Replacement metacharacters are backslash (\), & symbol, and n.
Backslashes are generally used to escape other metacharacters, but they are also used in the replacement string to include newlines.
For example, for the following line:
column1(tab)column2(tab)column3(tab)column4
Use the following replacement statement:
s/tab/
/2
Note that there must be no space after the backslash. This script produces the following result:
column1(tab)column2
column3(tab)column4
The "&" symbol is a metacharacter representing the range matched by the pattern, not the line being matched. For example, the following command:
s/UNIX/s-2&s0/g
Can replace the input line:
on the UNIX Operating System.
into:
on the s-2UNIXs0 Operating System.
The "&" symbol is particularly useful when the regular expression matches changes in words. It allows specifying a variable replacement string. References such as "See Section 1.4" or "See Section 12.9" should be in parentheses, like "(See Section 12.9)". The regular expression can match different combinations of numbers, so "&" can be used in the replacement string to enclose the matched content:
s/See Section *.*/(&)/
Here the "&" symbol is used to reference the entire matched content in the replacement string.
The n metacharacter is used to select any independent part of the matched string and call it back in the replacement string. In sed, escaped parentheses enclose any part of the regular expression and save it for callback. Up to 9 saves are allowed per line. For example, when a section number appears in a cross-reference and should be bolded:
s/(See Section )(*.*)/1fB2fp/
Another example:
$ cat test1
first:second
one:two
$ sed 's/(.*):(.*)/2:1/ test1
second:first
two:one

21．t
Usage:
]t
Test if a replacement was successfully executed within the addressed line range. If yes, transfer to the line with label flag (see b and :). If label is not given, control will transfer to the bottom of the script.

22．w
Usage:
]w file
Append the contents of the pattern space to file. This action occurs when the command is encountered, not when the pattern space content is output. A space must be kept between w and the filename. The maximum number of files that can be opened in the script is 10. If the file does not exist, this command will create a file. If the file exists, each time the script is executed, its content will be overwritten, and multiple write commands directly write output to the same file and append to the end of this file.

23．x
Usage:
]x
Swap the contents of the pattern space and the hold space.

24．y
Usage:
]y/abc/xyz/
Replace characters in string abc with corresponding characters in string xyz by position.

Last edited by 无奈何 on 2006-10-26 at 01:24 PM ]

Original link: http://phi.sinica.edu.tw/aspac/reports/96/96005/
SED Manual
Institute of Mathematics, Academia Sinica
ASPAC Project
aspac@phi.sinica.edu.tw
Technical Report: 96005
December 1, 1996
Version:1.0

--------------------------------------------------------------------------------

Table of Contents:

Copyright Notice

1. Introduction
When to Use sed
Where to Get sed
What sed Can Do
How sed Works

Using sed
Executing Edit Commands on the Command Line
sed's Edit Commands
Address Parameter Notation
Function Parameters
Executing Edit Commands in a File
Editing Multiple Files
Controlling Output

Examples
Substituting Data in Files
Moving Data in Files
Deleting Data in Files
Searching for Data in Files

Introducing Function Parameters
s
d
a
i
c
p
l
r
w
y
!
n
q
=
#
N
D
P
h
H
g
G
x
b
t

Appendix A: Common Regular Expressions

Appendix B: Acceptance of Special Characters in Regular Expressions by sed in HP-UX Release 9.01 and SunOS 5.4
References
Notes

--------------------------------------------------------------------------------

Introduction

--------------------------------------------------------------------------------

1.Introduction
Sed (Stream EDitor) is an editor on UNIX systems that automates the editing work, allowing users not to directly edit the data. Users can use more than 20 different function parameters provided by sed to combine (Note ) them to complete different editing actions. In addition, since sed edits files line by line, it is also a line editor.
Generally, sed is most commonly used to edit files that require repeating certain editing actions continuously, such as replacing a certain string in a file with another string. Compared with general UNIX editors (interactive ones like vi, emacs) that modify files manually, using sed is more labor-saving. The following sections will introduce respectively:

When to Use sed
Where to Get sed
What sed Can Do
How sed Works
1.1 When to Use sed
When modifying a file, if you repeatedly perform certain editing actions, you can use sed to automatically perform these editing actions at once. For example, to change the sender's alias "Tom" to "John" in 1000 emails in the received file, you can simply execute a simple sed command on the command line to replace all "Tom" strings in the file with "John".
Moreover, when a file requires many different editing actions, sed can perform those different editing actions at once. For example, sed can delete all blank lines in a file at once, replace strings, and add text entered by the user to the sixth line of the file, etc.

1.2 Where to Get sed
Generally, the sed is attached to the general UNIX system itself. The versions of sed attached to different UNIX systems are also different. If the sed is not attached to the UNIX system you are using, you can obtain it through anonymous ftp to the following places:
phi.sinica.edu.tw:/pub/GNU/gnu
gete.sinica.edu.tw:/unix/gnu
ftp.edu.tw:/UNIX/gnu
ftp.csie.nctu.edu.tw:/pub/Unix/GNU
ftp.fcu.edu.tw: /pub3/UNIX/gnu
axp350.ncu.edu.tw:/Packages/gnu
leica.ccu.edu.tw :/pub2/gnu
mail.ncku.edu.tw :/pub/unix/gnu
bbs.ccit.edu.tw :/pub1/UNIX/gnu
prep.ai.mit.edu.tw:/pub/gnu

1.3 What sed Can Do
sed can delete (delete), change (change), append (append), insert (insert), merge, exchange data lines in a file, or read data from other files into the file, and can also substitute (substuite) strings in them, or convert (tranfer) letters in them, etc. For example, delete consecutive blank lines in a file into one line, replace the string "local" with "remote", convert the letter "t" to "T", merge the data of line 10 and line 11, etc.

1.4 How sed Works
Like other UNIX commands, sed reads the edited file from standard input and sends the result to standard output. The following figure shows that sed replaces the data line "Unix" with "UNIX",

In the figure, the upper standard input is the standard input, which is the place to read data; the standard output is the place to send the result; the two dashed squares below the middle sed box represent the workflow of sed. Among them, the left dashed square means that sed puts the standard input data into the pattern space, and the right dashed square means that sed sends the data after editing in the pattern space to the standard output.

In the dashed square, the two solid squares respectively represent the pattern space and the sed script. Among them, the pattern space is a buffer, which is the working place of sed; and the sed script represents a set of editing instructions to be executed.

In the figure, the "Unix" on the left dashed square is put into the pattern space from the standard input; then, in the right dashed square, sed executes the editing instruction s/Unix/UNIX/ in the sed script (Note ), the result "Unix" is replaced with "UNIX", and then "UNIX" is sent from the pattern space to the standard output.

In summary, when sed reads a line of data from the standard input and puts it into the pattern space, sed executes the editing on the data in the pattern space one by one according to the editing instructions of the sed script, and then sends the result in the pattern space to the standard output, and then reads the next line of data. This action is repeated until all data lines are read.

--------------------------------------------------------------------------------

Using sed

--------------------------------------------------------------------------------

Using sed
The sed command line can be divided into edit commands and file parts. Among them, the edit commands are responsible for controlling all editing work; the file part represents the file to be processed. The sed edit commands are composed of two parts: address and function. When executing, sed uses its address parameter to determine the object of editing; and uses its function parameter (Note ) to edit.
In addition, the sed edit commands can be executed not only on the command line but also in a file. The difference is that when executing on the command line, the option -e must be added before it; when in a file (Note ), only the option -f needs to be added before its file name. In addition, sed executes the edit commands in the order they are on the command line or in the file.

The following sections will introduce executing edit commands on the command line, sed edit commands, executing edit commands in a file, editing multiple files, and controlling sed output.

2.1 Executing Edit Commands on the Command Line
2.2 sed's Edit Commands
2.3 Executing Edit Commands in a File
2.4 Editing Multiple Files
2.5 Controlling sed Output
2.1. Executing Edit Commands on the Command Line
When the edit command (refer to ) is executed on the command line, the option -e must be added before it. The command format is as follows:
sed -e 'edit command 1' -e 'edit command 2' ... file

Among them, all edit commands are immediately after the option -e and are placed between two " ' " special characters. In addition, the execution of the edit commands on the command line is from left to right.

When there are not many edit commands, users usually execute them directly on the command line. For example, to delete the data from line 1 to 10 in yel.dat and replace the string "yellow" with "black" in the remaining text. At this time, the edit commands can be executed directly on the command line, and the command is as follows:

sed -e '1,10d' -e 's/yellow/black/g' yel.dat

In the command, the edit command '1,10d' (Note ) deletes the data from line 1 to 10; the edit command 's/yellow/black/g' (Note ) replaces the string "yellow" with "black".
2.2 sed's Edit Commands
The format of the sed edit command is as follows:
]function

Among them, the address parameters address1 and address2 are line numbers or regular expression strings, representing the data lines to be edited; the function parameter function is the built-in function of sed, representing the editing action to be executed.
The following two sections will carefully introduce the notation of the address parameter and which function parameters are available for selection.

2.2.1 Address Parameter Notation
In fact, the address parameter notation is just to represent the data lines to be edited by their line numbers or strings in them. The following examples are used to illustrate (the command uses the function parameter d (refer to ) as an example):
To delete the data in line 10 of the file, the command is 10d.
To delete the data line containing the string "man", the command is /man/d.
To delete the data from line 10 to line 200 in the file, the command is 10,200d.
To delete from line 10 to the data line containing the string "man" in the file, the command is 10,/man/d.
Next, according to the content and number of address parameters, the notation of the address parameter in the command is fully explained (also taking the function parameter d as an example).
Content of address parameters:
The address is a decimal number: this number represents the line number. When the command is executed, the editing action indicated by the function parameter is executed on the data that matches this line number. For example, to delete the data in line 15 of the data file, the command is 15d (refer to ). And so on, for example, to delete the data in line m of the data file, the command is md.

Address is a regular expression (refer to ):
When there is a string in the data line that matches the regular expression, the editing action indicated by the function parameter is executed. In addition, "/" must be added before and after the regular expression. For example, the command is /t.*t/d, which means deleting all data lines containing two "t" letters. Among them, "." represents any character; "*" represents that the previous character can be repeated any number of times, and they are combined with ".*" to represent any string between two "t" letters.

Number of address parameters: In the command, when there is no address parameter, it means that all data lines execute the editing indicated by the function parameter; when there is only one address parameter, it means that only the data line that matches the address is edited; when there are two address parameters, such as address1,address2, it means editing the data area, where address1 represents the starting data line and address2 represents the ending data line. For the above content, the following examples are used for specific explanation.

For example, the command is

d

It means deleting all data lines in the file.
For example, the command is

5d

It means deleting the data in line 5 of the file.
For example, the command is

1,/apple/d

It means deleting the data area from the first line of the file to the data line containing the string "apple".
For example, the command is

/apple/,/orange/d

It means deleting the data area from the data line containing the string "apple" to the data line containing the string "orange" in the file
2.2.2 What Function Parameters Are Available
The following table introduces the functions of all sed function parameters (refer to ).
Function Parameter Function
: label Establish a position for mutual reference of instructions in the script file.
# Establish a comment
{ } Collect instructions with the same address parameter.
! Do not execute the function parameter.
= Print the line number (line number) of the data.
a\ Add data entered by the user.
b label Branch the executed instruction to the reference position established by :.
c\ Replace data with data entered by the user.
d Delete data.
D Delete the data before the first newline character \ in the pattern space.
g Copy data from the hold space.
G Add data from the hold space to the pattern space.
h Copy data from the pattern space to the hold space.
H Add data from the pattern space to the hold space.
l Print nonprinting characters in the l data in ASCII code.
i\ Insert and add the data line entered by the user.
n Read the next data.
N Add the next data to the pattern space.
p Print data.
P Print the data before the first newline character \ in the pattern space.
q Exit the sed edit.
r Read the content of another file.
s Substitute strings.
t label First execute a substitution edit command, and if the substitution is successful, jump the edit command to : label to execute.
w Write data to another file.
x Swap the contents of the hold space and the pattern space.
y Transform characters.
Although sed only has the above-mentioned few basic function parameters with editing functions, through the cooperation between the address parameters in the instruction and between instructions, sed can also complete most editing tasks.
2.3 Executing Edit Commands in a File
When there are too many commands to be executed and it is very messy to write on the command line, you can sort and store these commands in a file (for example, the file name is script_file), and use the option -f script_file to let sed execute the edit commands in script_file. The command format is as follows:
sed -f script_file file

Among them, the order of executing the edit commands in script_file is from top to bottom. For example, the example in the previous section can be changed to the following command:
sed -f ysb.scr yel.dat

Among them, the content of ysb.scr is as follows:
1,10d
s/yellow/black/g

In addition, on the command line, options -e and -f can be mixed, and the order of sed executing commands is still from left to right on the command line. When executing to the edit commands in the file after -f, it is executed from top to bottom.

2.4 Editing Multiple Files
In the sed command line, multiple files can be edited at one time, and they follow after the edit commands. For example, to replace the string "yellow" with "blue" in the files white.dat, red.dat, and black.dat, the command is as follows:
sed -e 's/yellow/blue/g' white.dat red.dat black.dat

When the above command is executed, sed executes the edit command s/yellow/blue/ (refer to to replace the string in order of white.dat, red.dat, black.dat.

2.5. Controlling Output
The option -n (Note ) on the command line means that the output is controlled by the edit command. From the content of the previous chapter, it is known that sed will "automatically" send data from the pattern space to the standard output file. But with the option -n, sed can change this "automatic" action to "passive" to be determined by the edit commands it executes (Note ) whether the result is output.
It can be seen from the above that the option -n must be used together with the edit command, otherwise the result cannot be obtained. For example, to print the data line containing the string "white" in the white.dat file, the command is as follows:

sed -n -e '/white/p' white.dat

In the above command, the option -n and the edit command /white/p (refer to ) cooperate together to control the output. Among them, the option -n transfers the output control to the edit command; /white/p prints the data line containing the string "white" on the screen.
--------------------------------------------------------------------------------

3. Examples

--------------------------------------------------------------------------------

3. Examples
Generally, in the process of actually using the editor, it is often necessary to perform actions such as substituting strings in files, moving, deleting, and searching for data lines. Of course, general interactive editors (such as vi, emacs) can all do the above functions, but when there are a large number of the above editing requirements for a file, using them to edit is very inefficient. This chapter will use examples to illustrate how to use sed to automatically perform these editing functions. In addition, in the examples of this chapter, the requirements of the file are described in the following way:
Replace ... data in the file with ... (action)

In this way, the purpose is to quickly convert them into edit commands. Among them, the part of "... data" is converted into the address parameter representation in the instruction; the part of "execute ... action" is converted into the function parameter representation. In addition, when "execute ... action" is to be represented by several function parameters, these function parameters can be collected by using "{ " and " }" (Note ). The instruction form is as follows:
address parameter{
function parameter 1
function parameter 2
function parameter 3
.
:
}

The above instruction means that for the data that matches the address parameter, the actions represented by function parameter 1, function parameter 2, function parameter 3... are executed in sequence. The following sections respectively give examples to illustrate the commands of sed for substituting data, moving, deleting data, and searching for data.
3.1 Substituting Data in Files
3.2 Moving Data in Files
3.3 Deleting Data in Files
3.4 Searching for Data in Files
3.1 Substituting Data in Files
Sed can substitute strings, data lines, and even data areas in files. Among them, the function parameter s (refer to ) in the instruction representing substituting strings; the function parameter c (refer to ) in the instruction representing substituting data lines or data areas. The above situations are illustrated by the following three examples. The above situations are illustrated by the following three examples.
Example 1. Replace the string "phi" in the data line containing the string "machine" in the file with the string "beta". The command line is as follows:
sed -e '/machine/s/phi/beta/g' input.dat (from now on, the file name is input.dat)

Example 2. Replace the data in line 5 of the file with the sentence "Those who in quarrels interpose, must often wipe a bloody nose.". The command line is as follows
sed -e '5c\
Those must often wipe a bloody nose.
' input.dat

Example 3. Replace the data area from line 1 to 100 in the file with the following two lines of data:
How are you?
data be deleted!

Then the command line is as follows
sed -e '1,100c\
How are you?\
data be deleted!
' input.dat

3.2 Moving Data in Files
Users can use the hold space in sed to temporarily store the data being edited, use the function parameter w (refer to ) to move the file data to another file for storage, or use the function parameter r (refer to ) to move the content of another file to the file. The hold space is a register used by sed to temporarily store the data in the pattern space. When sed executes the function parameters h and H (refer to ), it will temporarily store the pattern space data in the hold space; when executing the function parameters x, g, G (refer to ), it will take the temporarily stored data to the pattern space. The following three examples are used to illustrate.
Example 1. Move the first 100 data in the file to the output after line 300 in the file. The command line is as follows:

sed -f mov.scr file

The content of mov.scr is
1,100{
H
d
}
300G

Among them,

1,100{
H
d
}

It means that the first 100 data in the file are first stored (refer to ) in the hold space and then deleted; the instruction 300G (refer to ) means that the data in the hold space is added after the data in line 300 of the file and output.
Example 2. Move the data line containing the string "phi" in the file to be stored in the mach.inf file. The command line is as follows:
sed -e '/phi/w mach.inf' file

Example 3. Move the content of the mach.inf file to the data line containing the string "beta" in the file. The command line is as follows:
sed -e '/beta/r mach.inf' file

In addition, since sed is a stream (refer to ) editor, theoretically, the data of the output file cannot be moved back for editing.

3.3 Deleting Data in Files
Because sed is a line editor, sed can easily delete individual data lines or entire data areas. Generally, the function parameters d (refer to ) or D (refer to ) are used to represent. The following two examples are used to illustrate.
Delete all blank lines in the file. The command line is
sed -e '/^$/d' file

Regular expression (Note ), ^$ means a blank line. Among them, ^ restricts that the following string must be at the beginning of the line; $ restricts that the preceding string must be at the end of the line.
Delete consecutive blank lines in the file and delete them to become one line. The command line is
sed -e '/^$/{
N
/^$/D
}' file

Among them, the function parameter N (refer to ) means that the data line below the blank line is added to the pattern space. The function parameter /^$/D means that when the added one is a blank line, the first blank line is deleted, and the remaining blank lines are then re-executed once. The instruction is re-executed once, and a blank line is deleted. This is repeated until the blank line is followed by a non-blank line, so that only one blank line remains after consecutive blank lines and is output.
3.4 Searching for Data in Files
Sed can perform functions similar to the UNIX command grep. In theory, regular expressions (refer to ) can be used. For example, to output the data line containing the string "gamma" in the file. Then the command line is as follows:
sed -n -e '/gamma/p' file

However, sed is a line editor, and its search is basically line-based. Therefore, when some strings are split into two parts due to line breaks, the general method is not feasible. At this time, the data must be searched by merging two lines. The situation is as follows in the following example:
Example. Output the data containing the string "omega" in the file. The command line is as follows

sed -f gp.scr file

The content of gp.scr is as follows:
/omega/b
N
h
s/.*\n//
/omega/b
g
D

In the above sed script (Note ), because the function parameter b forms a case statement structure similar to C language, sed can respectively handle the situations where the data contains the string "omega"; when the string "omega" is split into two lines; and when the data does not contain the string "omega". Next, according to the above three situations, the sed script is divided into the following three parts for discussion.
When the data contains "omega", execute the edit command
/omega/b

It means that when the data contains the string "omega", sed does not need to execute the following instructions on it anymore, and directly outputs it.
When the data does not contain "omega", execute the following edit commands
N
h
s/.*\n//
/omega/b

Among them, the function parameter N (refer to ) means that the next line of data is read so that the pattern space contains the previous and next two lines of data. The function parameter h (refer to ) means that the previous and next two lines of data in the pattern space are stored in the hold space. The function parameter s/.*\n// means that the previous and next two lines of data in the pattern space are merged (Note ) into one line. /omega/b means that if the merged data contains the string "omega", then the following instructions are not executed anymore, and this data is automatically output;

When the merged data still does not contain "omega", execute the following edit commands
g
D

Among them, the function parameter g (refer to ) means that the two lines of data before merging in the hold space are put back into the pattern space. The function parameter D (refer to ) means that the first line of data in the two lines of data is deleted, and the remaining line of data is made to re-execute the sed script. In this way, the strings in the data line or between lines can be searched completely.

--------------------------------------------------------------------------------

Introducing Function Parameters

--------------------------------------------------------------------------------

Introducing Function Parameters
This chapter will introduce all the function parameters provided by sed in the way of one function parameter per section, including

| s | d | a | i | c | p | l | r | w | y | ! | n | q | = | # | N | D | P | h | H | g | G | x | b | t |

In addition, in each section, the function of the function parameter is briefly introduced first, and then the format of the function parameter cooperating with the address parameter is explained, and the working situation of sed executing this function parameter is also described.
4.1 s
The function parameter s represents substituting (substitute) strings in the file. The instruction format is as follows:
] s/pattern/replacemen/
The following points are explained for the above format:

The function parameter s cooperates with up to two address parameters.
Regarding "s/pattern/replacement/" (Note ), the following points are explained:
pattern: it is a regular expression string. It represents the string to be replaced in the file.
replacement: it is a general string. But the following characters have special meanings:

&: represents the previous pattern string. For example
sed -e 's/test/& my car/' file name

In the instruction, & represents the pattern string "test". Therefore, after execution, "test" in the data file is replaced with "test my car".
\n: represents the string enclosed by the nth \( and \) (refer to ) in the pattern. For example
sed -e 's/\(test\) \(my\) \(car\)//' file name

In the instruction, \1 represents "test", \2 represents "my", and \1 represents "car" string. Therefore, after execution, "test my car" in the data file is replaced with "".
\: It can be used to restore the literal meaning of some special symbols (such as & and \) above, or to represent a line break.
flag: mainly used to control some substitution situations:
When flag is g, it means replacing all matching (match) strings.
When flag is the decimal number m, it means replacing the mth matching string in the line.
When flag is p, it means that after replacing the first matching pattern string, the data is output to the standard output file.
When flag is w wfile, it means that after replacing the first matching pattern string, it is output to the wfile file (if wfile does not exist, a file named wfile will be re-opened).
When there is no flag, the first matching pattern string in the data line is replaced with the replacement string.
delimiter: In "/pattern/replace/ ", "/" is used as a delimiter. In addition to blank (blank) and newline (newline), users can use any character as the delimiter. For example, the following edit command
s#/usr#/usr1#g

In the above command, \verb|#| is the delimiter. If "/" is used as the delimiter, sed will treat "/" in the pattern and replacement as the delimiter and an error will occur.
Example:
Topic: Replace the string "1996" in the input.dat file (if not specified specially later, it is assumed that the file name is input.dat) with "1997", and store the data lines that have been replaced in the year97.dat file.
Description: Use the function parameter s to instruct sed to replace the string "1996" with "1997", and use the flag w in the s argument to instruct sed to store the replaced data lines in the year97.dat file.
sed command line:
sed -e 's/1996/1997/w year97.dat' input.dat

4.2 d
The function parameter d means deleting the data line, and the instruction format is as follows:

] d

The following points are explained for the above format:

The function parameter d cooperates with up to two address parameters.
The situation when sed executes the delete action is as follows:
Delete the data in the pattern space that matches the address parameter.
Read the next data into the pattern space.
Re-execute the sed script.
Example: Refer to section 3.3.
4.3 a
The function parameter a means adding data to the file. The instruction format is as follows:

a\ data entered by the user

The following points are explained for the above format:

The function parameter a cooperates with up to one address parameter.
The function parameter a is followed by the "\" character to indicate the end of this line, and the data entered by the user must be entered from the next line. If the data exceeds one line, "\" must be added at the end of each line.
The situation when sed executes the add action is as follows: After the data in the pattern space is output, sed then outputs the data entered by the user.
Example:
Topic: Add "Multitasking System" after the data line containing the string "UNIX". Assume that the content of input.dat is as follows:
UNIX

Description: Use the function parameter a to add the input data after the data line containing the string "UNIX".
The sed command line is as follows:
sed -e '/UNIX/a\
Multitasking System
' input.dat

After executing the above command, the output result is as follows:
UNIX
Multitasking System

4.4 i
The function parameter i means inserting data into the file. The instruction format is as follows:
i\ data entered by the user

The following points are explained for the above format:

The function parameter i cooperates with up to one address parameter.
The function parameter i is followed by the "\" character to indicate the end of this line, and the data entered by the user must be entered from the next line. If the data exceeds one line, "\" must be added at the end of each line.
The situation when sed executes the insert action is as follows: Before the data in the pattern space is output, sed first outputs the data entered by the user.
Example:
Topic: Insert "The copyright of the article belongs to the Academia Sinica" before the data line containing "Director: Lee Yuan-tseh" in the input.dat file. Assume that the content of input.dat is as follows:
Director: Lee Yuan-tseh

Description: Use the function parameter i to insert the data line "The copyright of the article belongs to the Academia Sinica" before the data line containing "Director: Lee Yuan-tseh".
The sed command line is as follows:
sed -e '/Director: Lee Yuan-tseh/i\
The copyright of the article belongs to the Academia Sinica
' input.dat

After executing the above command, the output is as follows:
The copyright of the article belongs to the Academia Sinica
Director: Lee Yuan-tseh

4.5 c
The function parameter c means changing the data in the file. The format is as follows:
]c\ data entered by the user

The following points are explained for the above format:

The function parameter c cooperates with up to two address parameters.
The function parameter c is followed by the "\" character to indicate the end of this line, and the data entered by the user must be entered from the next line. If the data exceeds one line, "\" must be added at the end of each line.
The situation when sed executes the change action: When the data in the pattern space is output, sed changes it to the data entered by the user.
Example: Refer to Example 2 and 3 in section 3.1.
4.6 p
The function parameter p means printing data. The instruction format is as follows:
] p

The following points are explained for the above format:

The function parameter p cooperates with up to two address parameters.
The situation when sed executes the print action is as follows: sed copies a copy of the content of the pattern space to the standard output file.
Example: Refer to the content at the beginning of section 3.4.
4.7 l
The function parameter l, in addition to listing the nonprinting characters in the data in ASCII code, is the same as the function parameter p. For example, print the ^







.
4.8 r
The function parameter r means reading the content of another file into the file. The instruction format is as follows:

r file name

The following points are explained for the above format:

The function parameter r cooperates with up to one address parameter.
In the instruction, there can only be one space between the function parameter r and the file name.
The situation when sed executes the read action is as follows: After the data in the pattern space is output, sed reads the content of the other file and outputs it. When the other file does not exist, sed still executes other instructions without any error message.
Example: Refer to Example 3 in section 3.1.
4.9 w
The function parameter w means writing the file to another file. The instruction format is as follows:

] w file name

The following points are explained for the above format:

The function parameter w cooperates with up to two address parameters.
In the instruction, there can only be one space between the function parameter w and the file name.
The situation when sed executes the write action is as follows: Write the data in the pattern space to another file. When writing data, it will overwrite (overwrite) the data in the original file. In addition, when the other file does not exist, sed will recreate (creat) it.
Example: Refer to Example 2 in section 3.1.
4.10 y
The function parameter y means converting characters in the data. The instruction format is as follows:

]y /xyz.../abc.../

The following points are explained for the above format:

The function parameter cooperates with up to two address parameters.
In the instruction, /abc.../xyz.../ (x, y, z, a, b, c represent certain characters) is the argument of y. Among them, the number of characters in abc... and xyz... must be the same.
When sed executes the conversion, the a character in the data in the pattern space is converted to the x character, the b character is converted to the y character, the c character is converted to the z character, etc.
Example:
Topic: Convert the lowercase letters in the input.dat file to uppercase. Assume that the content of input.dat is as follows:
Sodd's Second Law:
Sooner or later, the worst possible set of
circumstances is bound to occur.

Description: Use the function parameter y to instruct sed to convert the case of letters.
The sed command line is as follows:
sed -e '
y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/
' input.dat

After executing the above command, the output result is as follows:
SODD'S SECOND LAW:
SOONER OR LATER, THE WORST POSSIBLE SET OF
CIRCUMSTANCES IS BOUND TO OCCUR.

4.11 !
The function parameter ! means not executing the function parameter. When there is the following instruction,

] ! function parameter

It means that the data that matches the address parameter does not execute the function parameter. For example, delete all data lines except those containing the string "1996", then execute the following command

sed -e '/1996/!d' input.dat

4.12 n
The function parameter n means reading the next line of data. The instruction format is as follows:

] n

The following points are explained for the above format:

The function parameter n cooperates with up to two address parameters.
The situation when sed executes the action of reading the next line is as follows:
Output the data in the pattern space.
Read the next data into the pattern space.
Execute the next edit command.
Example (can be compared with the example in ):
Topic: Output the even-numbered line data in the input.dat file. Assume that the content of input.dat is as follows:
The
UNIX
Operation
System

Description: On the command line
With the option -n, transfer the output control right (refer to ) to the instruction.
Use the function parameter n to replace the data line (odd-numbered line) in the pattern space with the next line of data (even-numbered line).
Use the function parameter p to output the data (even-numbered line) in the pattern space.
Finally, the output only has the original even-numbered line data.
The sed command line is as follows:
sed -n -e 'n' -e 'p' infro.dat

After executing the above command, the output result is as follows:
UNIX
System

4.13 q
The function parameter q means jumping out of sed. The instruction format is as follows:

q

The following points are explained for the above format:

The function parameter q cooperates with up to one address parameter.
When sed executes the jump action, it stops inputting the pattern space data and stops sending the data to the standard output file.
Example:
Topic: Execute the edit commands in script_file on the file, unless the string "Linux" is encountered.
Description: No matter what instructions are in script_file, the user only needs to use the instruction /Linux/q on the command line, and the function parameter q will force sed to jump out when encountering "Linux".
The sed command line is as follows:
sed -e '/Linux/q' -f script_file input.dat

4.14 =
The function parameter = means printing the line number of the data. The instruction format is as follows:

] =

The following points are explained for the above format:

The function parameter = cooperates with up to two address parameters.
When executed, the line number will be output before the data is output.
Example:
Topic: Print the line number of the data in the input.dat file. Assume that the content of input.dat is as follows:
The UNIX
Operating System

Description: Use the function parameter = to print the line number of the data.
The sed command line is as follows:
sed -e '=' input.dat

After executing the above command, the output result is as follows:
1
The UNIX
2
Operating System

4.15 #
In the script file, the text after the function parameter # is a comment. When the comment text exceeds multiple lines, the line breaks must be separated by the "\" line break character.

4.16 N
The function parameter N means adding the next data in the pattern space. The instruction format is as follows:

] N

The following points are explained for the above format:

The function parameter N cooperates with up to two address parameters.
When sed executes, the next line of data is read and added to the pattern space, and the data lines are separated by the embedded newline character. In addition, when substituting, the newline character can be matched with \n.
Example:
Topic: Merge the following two lines of data. Assume that the content of input.dat is as follows:
The UNIX
Operating System

Description: First use the function parameter N to place the two lines of data in the pattern space, and then use the function parameter s/\n/ / to replace the separator \n between the two lines of data with a space, so that the two lines of data become one line output.
The sed command line is as follows:
sed -e 'N' -e 's/\n/ /' input.dat

After executing the above command, the output result is as follows:
The UNIX Operating System

4.17 D
The function parameter D means deleting the first line of data in the pattern space. The instruction format is as follows:

D

The following points are explained for the above format:

The function parameter D cooperates with up to two address parameters.
The comparison between the function parameter D and d is as follows:
When there is only one line of data in the pattern space, D and d have the same effect.
When there are multiple lines of data in the pattern space
D means only deleting the first line of data in the pattern space; d deletes all.
D means that after execution, the pattern space does not add the next data, and the remaining data is re-executed the sed script; d reads the next line and then executes the sed script.
Example: Refer to the second example in section 3.3.
4.18 P
The function parameter P means printing the first line of data in the pattern space. The instruction format is as follows:

P

The following points are explained for the above format:

The function parameter P cooperates with up to two address parameters.
P is the same as p except that the number of data lines in the facing pattern space is different.
Example (can be compared with the example in ):
Topic: Output the odd-numbered line data in the input.dat file. Assume that the content of input.dat is as follows:
The
UNIX
System

Description: On the command line
With the option -n, transfer the output control right (refer to ) to the instruction.
Use the function parameter N to add the even-numbered line to the odd-numbered line in the pattern space.
Use the function parameter P to output the first line (odd-numbered line) in the pattern space.
After the odd-numbered line is output, the remaining data line (even-numbered line) in the pattern space is abandoned and not output. Finally, the output only has the original odd-numbered line data.
The sed command line is:
sed -n -e 'N' -e 'P' infro.dat

After executing the above command, the output result is as follows:
The
System

4.19 h
The function parameter h means temporarily storing the data of the pattern space in the hold space. The instruction format is as follows:

] h

The following points are explained for the above format:

The function parameter h cooperates with up to two address parameters.
When sed executes the temporary storage action, it will overwrite (overwrite) the original data in the hold space.
When sed executes all, the data in the hold space will be automatically cleared.
Example: Refer to the example in section 3.4.
4.20 H
The only difference between the function parameter H and h is that when sed executes h, the data overwrites (overwrite) the original data in the hold space, while H, the data is "appended (append)" after the original data in the hold space. For the example, please refer to Example 1 in section 3.2.
4.21 g
The function parameter g means the opposite action of the function parameter h, which means putting the data in the hold space back into the pattern space. The instruction format is as follows:

g

The function parameter g cooperates with up to two address parameters.
When sed executes the back action, the data overwrites (overwrite) (Note ) the original data in the pattern space.
Example: Refer to the example in section 3.4.
4.22 G
The only difference between the function parameter G and g is that when sed executes g, the data overwrites (overwrite) the original data in the pattern space, while G, the data is "appended (append)" after the original data in the pattern space. For an example, please refer to Example 1 in section 3.2.
4.23 x
The function parameter x means exchanging the data in the hold space and the pattern space. The instruction format is as follows:

] x

The function parameter x mostly cooperates with other function parameters that process the hold space. For example, replace the data in line 1 of the input.dat file with the data in line 3. At this time, use the function parameters h and x to cooperate. Among them, use the function parameter h to store the first data in the hold space; when the data in line 3 appears in the pattern space, use the function parameter x to exchange the contents of the hold space and the pattern space. In this way, the data in line 3 is replaced by the first data. The command line is as follows:

sed -e '1h' -e '3x' input.dat

4.24 b、:label
The function parameter : and the function parameter b can establish a function similar to the GOTO instruction in the BASIC language in the sed script. Among them, the function parameter : establishes a mark; the function parameter b branches the next executed instruction to the mark for execution. The cooperation between the function parameter : and b in the script file is as follows

.
.
.
Edit command m1
:Mark
Edit command m2
.
.
.
]b

Among them, when sed executes to the instruction ]b , if the data in the pattern space matches the address parameter, sed will branch the next executed position to the mark set by :Mark (Note ), that is, execute from "Edit command m2" again. In addition, if there is no mark after the function parameter b in the instruction, sed will branch the next executed instruction to the end of the script file. Using this can make the sed script have a case statement structure similar to C language.
Example:
Topic: Repeat the first letter of the data line in the input.dat file 40 times. Assume that the content of input.dat is as follows:

A
B
C

Description: Use the instructions b p1 and :p1 to form a loop (loop) to execute the action of increasing letters, and at the same time, when 40 letters appear, use the instruction b to jump out of the loop. The following takes the first line of data "A" in the file as an example to describe how it continuously adds 39 more "A"s in the same line:
Use the instruction s/A/AA/ (refer to section4.1) to replace "A" with "AA".
Use the instructions b p1 and :p1 to form a loop (loop), which aims to repeatedly execute the above action. Each time the loop is executed, the "A" on the data line will be one more. For example, the data line becomes "AA" in the first loop, and becomes "AAA" in the second loop...
Use the instruction \{40\}/b (Note ) as the condition to stop the loop. When there are 40 consecutive A's appearing in the data line, the function parameter b will jump the executed instruction to the end and stop editing this line.
Similarly, the same way is executed for other data lines.
The sed command line is as follows:
sed -e '{
:p1
/A/s/A/AA/
/B/s/B/BB/
/C/s/C/CC/
/\{40\}/b
b p1
}' input.dat

4.25 t
Basically, the function parameter t is similar to the function parameter b in function, except that before executing the branch of t, it will first test whether the previous substitution instruction has been successfully substituted. The situation in the script file is as follows:
.
.
.
Edit command m1
:Mark
Edit command m2
.
.
.
s/.../.../
]t
Edit command m3

Among them, the difference from the function parameter b is that when executing the function parameter t branch, it will first check whether the previous substitution instruction is successful. If it is successful, execute the branch; if it is not successful, do not branch, and continue to execute the next edit command, such as the above edit command m3.
Example:
Topic: Replace A1 with C1, C1 with B1, and B1 with A1 in the input.dat file. The content of input.dat is as follows:
Code
B1
A1
B1
C1
A1
C1

Description: All data lines in the input.dat file only need to execute a substitution action, but to avoid the data being substituted multiple times, the function parameter t is used in the sed script to form a case statement structure similar to C language, so that each line of data can immediately jump out of the substitution edit after being substituted once.
The sed command line is:
sed -e '{
s/A1/C1/
t
s/C1/B1/
t
s/B1/A1/
t
}' input.dat
--------------------------------------------------------------------------------
Common Regular Expressions

--------------------------------------------------------------------------------

Common Regular Expressions
Ordinary characters The regular expression composed of ordinary characters has the same meaning as the literal meaning of the original string.
^string Restrict that the string must appear at the beginning of the line.
$string Restrict that the string must appear at the end of the line.
. Represents any character.
Character set, used to represent any one of all characters between the two brackets, such as represents any character other than all characters between the two brackets.
-& The character set can use "&" to specify the range of characters.
* Used to describe that the previous character (or character set) can be repeated any number of times.
\n Represents the embedded new line character (imbedded new line character).
\(...\) Use "\(" "\)" to enclose a part of the regular expression in the regular expression; later, "\1" can be used to represent the first part enclosed by "\(" "\)". If the regular expression uses "\(" "\)" several times to enclose different parts, then use "\1", "\2", "\3",... (up to "\9") in turn.
In addition, on different platforms, there are some different restrictions on regular expressions. For details, refer to appendix B.

--------------------------------------------------------------------------------

Notes

--------------------------------------------------------------------------------

Notes
Note 1.
It is the sed script that will be mentioned later.
Note 2.
The instruction s/Unix/UNIX/ means replacing "Unix" with "UNIX". Please refer to section 4.1.
Note 3.
There are more than 20 function parameters available for selection in the instruction.
Note 4.
This file is called the script file later.
Note 5.
In the edit command 1,10d, the address parameter is 1,10, so the data from line 1 to 10 executes the delete action specified by the function parameter d.
Note 6.
In the edit command s/yellow/black/g, since there is no address parameter, all data lines must execute the replacement action specified by the function parameter s/yellow/black/g. In the function parameter s/yellow/black/g, /yellow/black/g is the argument of s, which means replacing all "yellow" in the data line with "black".
Note 7.
The command format is as follows:
sed -n .. ..

Note 8.
These editing instructions may be one of p, l, s.
Note 9.
In some cases, the edit command can also be used instead of the function parameter. For example, Example 2 in section3.3.
Note 10.
Here, the sed script refers to the content of the gp.scr file. It means the edit command executed by sed this time.
Note 11.
This function parameter means replacing (removing) the newline mark between the two lines in the pattern space. Therefore, there is only one line of data in the pattern space.
Note 12.
/pattern/replacement/ is the argument of the function parameter s.
Note 13.
Note that at this time, although the data is put back into the pattern space, the content of the hold space remains unchanged.
Note 14.
Note that there must be no space between ":" and the mark.
Note 15.
The address parameter \{40\} means 40 A letters or 40 B letters or 40 C letters. Among them, means "A" or "B" or "C"; the following \{40\} means that there are 40 of the previous letters. For regular expressions, please refer to
Appendix A
.

--------------------------------------------------------------------------------

SED Manual

References

--------------------------------------------------------------------------------

References
``SED - A Non-interactive Text Editor '' Lee E.McMahon , AT&T Bell Laboratories Murray Hill,New Jersey 07947.
`` sed & awk'' Dale Dougherty , O'Reilly & Associates , Inc.1990.
``SunOs5.1 Editing Text Files'',Sun Microsystem,Inc.1992.
``HP 9000 computers -- Text Processing : User Guide'',Hewlett-Packard Company.1991.
`` Ye Daye. Introduction to SED, a Tool for Automatically Editing Files'', Bulletin of the Computing Center, Academia Sinica, Volume 12, Issue 2.

-------------------------------------------------------------------------
SED ONE-LINE SCRIPT QUICK REFERENCE (Unix Stream Editor) December 29, 2005
English Title: USEFUL ONE-LINE SCRIPTS FOR SED (Unix stream editor)
Original Title: HANDY ONE-LINERS FOR SED (Unix stream editor)
Organized by: Eric Pement - Email: pementenorthparkedu Version 5.5
Translator: Joe Hong - Email: hq00e126com
The latest (English) version of this document can be found at:

http://sed.sourceforge.net/sed1line.txt

http://www.pement.org/sed/sed1line.txt
Other language versions:
Chinese -
http://sed.sourceforge.net/sed1line_zh-CN.html
Czech -
http://sed.sourceforge.net/sed1line_cz.html
Dutch -
http://sed.sourceforge.net/sed1line_nl.html
French -
http://sed.sourceforge.net/sed1line_fr.html
German -
http://sed.sourceforge.net/sed1line_de.html
Portuguese -
http://sed.sourceforge.net/sed1line_pt-BR.html
Text Spacing:
--------
# Add a blank line after each line
sed G
# Delete all original blank lines and add a blank line after each line.
# This will result in exactly one blank line after each line in the output.
sed '/^$/d;G'
# Add two blank lines after each line
sed 'G;G'
# Delete all blank lines produced by the first script (i.e., delete all even lines)
sed 'n;d'
# Insert a blank line before lines matching the pattern "regex"
sed '/regex/{x;p;x;}'
# Insert a blank line after lines matching the pattern "regex"
sed '/regex/G'
# Insert a blank line before and after lines matching the pattern "regex"
sed '/regex/{x;p;x;G;}'
Numbering:
--------
# Number each line in the file (simple left-aligned). Here, a "tab" (see the description of '\t' usage at the end of this document) is used instead of spaces to align the edges.
sed = filename | sed 'N;s/\n/\t/'
# Number all lines in the file (line numbers on the left, text right-aligned).
sed = filename | sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /'
# Number all lines in the file, but only show line numbers for non-blank lines.
sed '/./=' filename | sed '/./N; s/\n/ /'
# Count the number of lines (simulates "wc -l")
sed -n '$='
Text Conversion and Substitution:
--------
# Unix environment: Convert DOS newlines (CR/LF) to Unix format.
sed 's/.$//' # Assuming all lines end with CR/LF
sed 's/^M$//' # In bash/tcsh, press Ctrl-M to be Ctrl-V
sed 's/\x0D$//' # ssed, gsed 3.02.80, and later versions
# Unix environment: Convert Unix newlines (LF) to DOS format.
sed "s/$/`echo -e \\\r`/" # Command used under ksh
sed 's/$'"/`echo \\\r`/" # Command used under bash
sed "s/$/`echo \\\r`/" # Command used under zsh
sed 's/$/\r/' # gsed 3.02.80 and later versions
# DOS environment: Convert Unix newlines (LF) to DOS format.
sed "s/$//" # Method 1
sed -n p # Method 2
# DOS environment: Convert DOS newlines (CR/LF) to Unix format.
# The following script is only valid for UnxUtils sed 4.0.7 and later versions. To identify the UnxUtils version of sed, you can check for its unique "--text" option. You can use the help option ("--help") to see if there is a "--text" item to determine if it is the UnxUtils version. Other DOS versions of sed cannot perform this conversion. But it can be achieved with "tr".
sed "s/\r//" infile >outfile # UnxUtils sed v4.0.7 or later
tr -d \r outfile # GNU tr 1.22 or later versions
# Delete leading "whitespace characters" (spaces, tabs) from each line
# To left-align
sed 's/^*//' # See the description of '\t' usage at the end of this document
# Delete trailing "whitespace characters" (spaces, tabs) from each line
sed 's/*$//' # See the description of '\t' usage at the end of this document
# Delete leading and trailing whitespace characters from each line
sed 's/^*//;s/*$//'
# Insert 5 spaces at the beginning of each line (shift the entire text 5 characters to the right)
sed 's/^/ /'
# Right-align all text to a width of 79 characters
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # 78 characters plus one final space
# Center all text to a width of 79 characters. In method 1, spaces are padded at the beginning and end of each line to center the text. In method 2, spaces are only padded in front of the text during the centering process, and eventually half of these spaces will be deleted. Also, no spaces are padded at the end of each line.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # Method 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' # Method 2
# Find the string "foo" in each line and replace the found "foo" with "bar"
sed 's/foo/bar/' # Replace only the first "foo" string in each line
sed 's/foo/bar/4' # Replace only the fourth "foo" string in each line
sed 's/foo/bar/g' # Replace all "foo" with "bar" in each line
sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # Replace the second-to-last "foo"
sed 's/\(.*\)foo/\1bar/' # Replace the last "foo"
# Only replace "foo" with "bar" if the string "baz" appears in the line
sed '/baz/s/foo/bar/g'
# Replace "foo" with "bar" and only replace if the string "baz" does not appear in the line
sed '/baz/!s/foo/bar/g'
# Replace "scarlet", "ruby", or "puce" with "red" regardless
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # Valid for most seds
gsed 's/scarlet\|ruby\|puce/red/g' # Only valid for GNU sed
# Reverse all lines, with the first line becoming the last line, and so on (simulates "tac").
# For some reason, HHsed v1.5 will delete blank lines in the file when using the following command
sed '1!G;h;$!d' # Method 1
sed -n '1!G;h;$p' # Method 2
# Reverse the characters in a line, with the first word becoming the last word, etc. (simulates "rev")
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'
# Combine every two lines into one line (similar to "paste")
sed '$!N;s/\n/ /'
# If the current line ends with a backslash "\", merge the next line to the end of the current line
# and remove the original line end backslash
sed -e :a -e '/\\$/N; s/\\\n//; ta'
# If the current line starts with an equal sign, merge the current line to the end of the previous line
# and replace the original line start "=" with a single space
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'
# Add commas to number strings to separate thousands, changing "1234567" to "1,234,567"
gsed ':a;s/\B\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/\(.*\)\(\{3\}\)/\1,\2/;ta' # Other seds
# Add commas to separate thousands for numerical values with decimal points and negative signs (GNU sed)
gsed -r ':a;s/(^|)(+)({3})/\1\2,\3/g;ta'
# Add a blank line after every 5 lines (add a blank line after lines 5, 10, 15, 20, etc.)
gsed '0~5G' # Only valid for GNU sed
sed 'n;n;n;n;G;' # Other seds
Selectively Display Specific Lines:
--------
# Display the first 10 lines of the file (simulates the behavior of "head")
sed 10q
# Display the first line of the file (simulates "head -1" command)
sed q
# Display the last 10 lines of the file (simulates "tail")
sed -e :a -e '$q;N;11,$D;ba'
# Display the last 2 lines of the file (simulates "tail -2" command)
sed '$!N;$!D'
# Display the last line of the file (simulates "tail -1")
sed '$!d' # Method 1
sed -n '$p' # Method 2
# Display the second-to-last line of the file
sed -e '$!{h;d;}' -e x # Outputs a blank line when there is only one line in the file
sed -e '1{$q;}' -e '$!{h;d;}' -e x # Displays the line when there is only one line in the file
sed -e '1{$d;}' -e '$!{h;d;}' -e x # Does not output when there is only one line in the file
# Only display lines matching the regular expression (simulates "grep")
sed -n '/regexp/p' # Method 1
sed '/regexp/!d' # Method 2
# Only display lines that "do not" match the regular expression (simulates "grep -v")
sed -n '/regexp/!p' # Method 1, corresponding to the previous command
sed '/regexp/d' # Method 2, similar syntax
# Find "regexp" and display the line before the matching line without showing the matching line
sed -n '/regexp/{g;1!p;};h'
# Find "regexp" and display the line after the matching line without showing the matching line
sed -n '/regexp/{n;p;}'
# Display the line containing "regexp" and its preceding and following lines, and add the line number of the line containing "regexp" before the first line (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h
# Display lines containing "AAA", "BBB", or "CCC" (in any order)
sed '/AAA/!d; /BBB/!d; /CCC/!d' # The order of the strings does not affect the result
# Display lines containing "AAA", "BBB", and "CCC" (fixed order)
sed '/AAA.*BBB.*CCC/!d'
# Display lines containing "AAA", "BBB", or "CCC" (simulates "egrep")
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # Most seds
gsed '/AAA\|BBB\|CCC/!d' # Valid for GNU sed
# Display paragraphs containing "AAA" (paragraphs are separated by blank lines)
# HHsed v1.5 must add "G;" after "x;", and the next three scripts are like this
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'
# Display paragraphs containing "AAA", "BBB", and "CCC" strings (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'
# Display paragraphs containing any one of the strings "AAA", "BBB", "CCC" (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d' # Only valid for GNU sed
# Display lines containing 65 or more characters
sed -n '/^.\{65\}/p'
# Display lines containing fewer than 65 characters
sed -n '/^.\{65\}/!p' # Method 1, corresponding to the above script
sed '/^.\{65\}/d' # Method 2, a simpler method
# Display part of the text - from the line containing the regular expression to the end of the last line
sed -n '/regexp/,$p'
# Display part of the text - specify a line number range (from line 8 to line 12, including lines 8 and 12)
sed -n '8,12p' # Method 1
sed '8,12!d' # Method 2
# Display line 52
sed -n '52p' # Method 1
sed '52!d' # Method 2
sed '52q;d' # Method 3, more efficient when processing large files
# Display every 7th line starting from line 3
gsed -n '3~7p' # Only valid for GNU sed
sed -n '3,${p;n;n;n;n;n;n;}' # Other seds
# Display text between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p' # Case-sensitive way
Selectively Delete Specific Lines:
--------
# Display the entire document except the content between two regular expressions
sed '/Iowa/,/Montana/d'
# Delete adjacent duplicate lines in the file (simulates "uniq")
# Only keep the first line of duplicate lines, delete other lines
sed '$!N; /^\(.*\)\n\1$/!P; D'
# Delete duplicate lines in the file, regardless of adjacency. Note the cache size supported by the hold space, or use GNU sed.
sed -n 'G; s/\n/&&/; /^\(*\n\).*\n\1/d; s/\n//; h; P'
# Delete all lines except duplicate lines (simulates "uniq -d")
sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'
# Delete the first 10 lines of the file
sed '1,10d'
# Delete the last line of the file
sed '$d'
# Delete the last two lines of the file
sed 'N;$!P;$!D;$d'
# Delete the last 10 lines of the file
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # Method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # Method 2
# Delete lines that are multiples of 8
gsed '0~8d' # Only valid for GNU sed
sed 'n;n;n;n;n;n;n;d;' # Other seds
# Delete lines matching the pattern
sed '/pattern/d' # Delete lines containing pattern. Of course, pattern
# can be replaced with any valid regular expression
# Delete all blank lines in the file (the same effect as "grep '.' ")
sed '/^$/d' # Method 1
sed '/./!d' # Method 2
# Only keep the first line of multiple adjacent blank lines. And delete blank lines at the top and bottom of the file.
# (Simulates "cat -s")
sed '/./,/^$/!d' # Method 1, delete blank lines at the top of the file, allowing one blank line at the bottom to remain
sed '/^$/N;/\n$/D' # Method 2, allowing one blank line at the top to remain, and no blank line at the bottom to remain
# Only keep the first two lines of multiple adjacent blank lines.
sed '/^$/N;/\n$/N;//D'
# Delete all blank lines at the top of the file
sed '/./,$!d'
# Delete all blank lines at the bottom of the file
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # Valid for all seds
sed -e :a -e '/^\n*$/N;/\n$/ba' # The same as above, but only valid for gsed 3.02.*
# Delete the last line of each paragraph
sed -n '/^$/{p;h;};/./{x;/./p;}'
Special Applications:
--------
# Remove nroff marks from man pages. When using the 'echo' command under Unix System V or bash shell, the -e option may be required.
sed "s/.`echo \\\b`//g" # The outer double brackets are necessary (Unix environment)
sed 's/.^H//g' # In bash or tcsh, press Ctrl-V then Ctrl-H
sed 's/.\x08//g' # Hexadecimal representation used by sed 1.5, GNU sed, ssed
# Extract the header of a newsgroup or e-mail
sed '/^$/q' # Delete all content after the first blank line
# Extract the body of a newsgroup or e-mail
sed '1,/^$/d' # Delete all content before the first blank line
# Extract the "Subject" from the email header and remove the leading "Subject: "
sed '/^Subject: */!d; s///;q'
# Get the reply address from the email header
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'
# Get the email address. Further remove non-email address parts from the line of email header generated by the previous script. (See the previous script)
sed 's/ *(.*)//; s/>.*//; s/.*



*>//g;/zipup.bat
dir /b *.txt | sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat
Using SED: Sed accepts one or more editing commands and applies these commands sequentially after each line is read.
After reading the first line of input, sed applies all commands to it and then outputs the result. Then it reads the second line of input, applies all commands to it... and repeats this process. In the previous example, sed gets input from the standard input device (i.e., the command interpreter, usually in the form of pipe input). When one or more filenames are given as parameters on the command line, these files replace the standard input device as the input of sed. The output of sed will be sent to the standard output (display). Therefore:
cat filename | sed '10q' # Use pipe input
sed '10q' filename # The same effect, but without using pipe input
sed '10q' filename > newfile # Redirect the output to disk
To understand the usage instructions of sed commands, including how to use these commands through script files (instead of from the command line), please refer to "sed & awk" second edition, authors Dale Dougherty and Arnold Robbins (O'Reilly, 1997; http://www.ora.com), "UNIX Text Processing", authors Dale Dougherty and Tim O'Reilly (Hayden Books, 1987) or the tutorial written by Mike Arst - the compressed package is named "U-SEDIT2.ZIP" (available on many sites). To explore the potential of sed, one must have sufficient understanding of "regular expressions". Information about regular expressions can be found in "Mastering Regular Expressions" by Jeffrey Friedl (O'reilly 1997).
The man pages provided by the Unix system ("man") will also be helpful (try these commands "man sed", "man regexp", or look at the part about regular expressions in "man ed"), but the information provided by the man pages is relatively "abstract" - which is also the reason why it has been criticized. However, it is not a textbook for teaching beginners how to use sed or regular expressions, but rather a text reference for those familiar with these tools.
Bracket Syntax: The previous examples basically use single quotes ('...') instead of double quotes ("...") for sed commands because sed is usually used on Unix platforms. Under single quotes, the Unix shell (command interpreter) will not interpret and execute the dollar sign ($) and backquote (`...`). Under double quotes, the dollar sign will be expanded to the value of a variable or parameter, and the command in the backquote will be executed and replaced by the output result. In "csh" and its derived shells, when using an exclamation point (!), a backslash (\) must be added in front of it (like this: \!) to ensure that the above examples can run normally (including when using single quotes). The DOS version of Sed always uses double quotes ("...") instead of quotes to enclose commands.
Usage of '\t': To keep this document concise, we use '\t' in the script to represent a tab. However, most current versions of sed do not recognize the short form of '\t', so when entering the tab character in the command line for the script, you should directly press the TAB key to enter the tab character instead of entering '\t'. The following tools support '\t' as a regular expression character to represent a tab: awk, perl, HHsed, sedmod, and GNU sed v3.02.80.
Different Versions of SED: There are some differences between different versions of sed, and it can be imagined that there are differences in syntax between them. Specifically, most of them do not support using labels (:name) or branch commands (b, t) in the middle of editing commands, unless they are placed at the end. In this document, we try to use more portable syntax so that users of most versions of sed can use these scripts. However, the GNU version of sed allows using more concise syntax. Imagine the mood of the reader when seeing a long command:
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
The good news is that GNU sed can make the command more compact:
sed '/AAA/b;/BBB/b;/CCC/b;d' # Can even be written as
sed '/AAA\|BBB\|CCC/b;d'
In addition, please note that although many versions of sed accept commands like "/one/ s/RE1/RE2/" with a space before 's', some of these versions do not accept such commands: "/one/! s/RE1/RE2/". In this case, just remove the space in the middle.
Speed Optimization: When you need to improve the execution speed of the command for some reason (such as a large input file, slow processor or hard disk, etc.), you can consider adding an address expression in front of the substitution command ("s/.../.../") to improve the speed. For example:
sed 's/foo/bar/g' filename # Standard substitution command
sed '/foo/ s/foo/bar/g' filename # Faster speed
sed '/foo/ s//bar/g' filename # Shorthand form
When you only need to display the front part of the file or need to delete the content at the back, you can use the "q" command (quit command) in the script. When processing large files, this will save a lot of time. Therefore:
sed -n '45,50p' filename # Display lines 45 to 50
sed -n '51q;45,50p' filename # The same, but much faster
If you have other one-line scripts to share or you find errors in this document, please send an email to the author of this document (Eric Pement). Please remember to provide the version of sed you are using, the operating system on which the sed runs, and an appropriate description of the problem in the email. The one-line scripts referred to in this document refer to sed scripts with a command line length of 65 characters or less . The various scripts in this document were written or provided by the following authors:
Al Aab # Established the "seders" mailing list
Edgar Allen # Many aspects
Yiorgos Adamopoulos # Many aspects
Dale Dougherty # Author of "sed & awk"
Carlos Duarte # Author of "do it with sed"
Eric Pement # Author of this document
Ken Pizzini # Author of GNU sed v3.02
S.G. Ravenhall # Remove html tags script
Greg Ubben # Made many contributions and provided a lot of help
-------------------------------------------------------------------------
Note 1: In most cases, sed scripts of any length can be written in a single line (through the '-e' option and ';' sign) - as long as the command interpreter supports it, so the so-called one-line script here is not only about being in a single line but also has a length limit. Because the meaning of these one-line scripts does not lie in their being in a single line. But it is meaningful that users can easily use these compact scripts in the command line.

Last edited by 无奈何 on 2006-10-26 at 01:28 PM ]

Repost Note: Original link http://blog.chinaunix.net/u/13392/showart.php?id=133128
sed - Non-interactive Text Editor
Lee E. McMahon

Bell Laboratories
Murray Hill, New Jersey 07974

Translation: Han Chan Tui Shi

Translator's Statement: The translator makes no warranties regarding the translation, has no rights to the translation, and assumes no responsibilities or obligations.
Original: http://cm.bell-labs.com/7thEdMan/vol2/sed

Abstract
sed is a non-interactive line editor that runs on the UNIX? operating system. sed is designed to function in the following three situations:

1) Editing files that are too large for comfortable interactive editing.
2) Editing files of any size when the editing commands are too complex to type in interactive mode.
3) Efficiently performing multiple 'global' editing functions in a single pass over the input.
This memo is a manual for sed users.

August 15, 1978


--------------------------------------------------------------------------------

Table of Contents
Introduction
1. Overall Operation
1.1. Command-line Flags
1.2. Order of Application of Editing Commands
1.3. Pattern Space
1.4. Examples
Examples
2. Addresses: Selecting Lines to Edit
2.1. Line Number Addresses
2.2. Context Addresses
2.3. Number of Addresses
Examples
3. Functions
3.1. Line-oriented Functions
Examples
3.2. Substitution Functions
Examples
3.3. Input/Output Functions
Examples
3.4. Multi-line Input Functions
3.5. Save and Retrieve Functions
Examples
3.6. Control Flow Functions
3.7. Miscellaneous Functions
References

--------------------------------------------------------------------------------

Introduction
sed is a non-interactive line editor designed to function in the following three situations:

1) Editing files that are too large for comfortable interactive editing. 2) Editing files of any size when the editing commands are too complex to type in interactive mode. 3) Efficiently performing multiple 'global' editing functions in a single pass over the input.

Since only certain lines of input are kept in memory at a time and no temporary files are used, the effective size of the file that can be edited is limited only by the requirement that input and output coexist in secondary storage simultaneously.

Complex editing scripts can be built separately and used as command files for sed. For complex edits, this saves significant typing and the resulting errors. Running sed from a command file is more efficient than any interactive editor the author is aware of, even including editors that can be driven by pre-written scripts.

The fundamental loss compared to interactive editors is the lack of relative addressing (since operations are line-by-line) and the lack of immediate verification that commands are running as expected.

sed is a direct descendant of the UNIX editor ed. Due to differences between interactive and non-interactive operation, there have been significant changes between ed and sed; even regular users of ed will often be surprised (and possibly annoyed) if they use sed without reading the sections 2 and 3 of this document. The most significant family resemblance between the two editors is the type of pattern ('regular expression') they recognize; the code for matching patterns can be copied almost unchanged from ed's code, and the description of regular expressions in section 2 is copied almost unchanged from the UNIX Programmer’s Manual. (Both the code and the description were written by Dennis M. Ritchie).


--------------------------------------------------------------------------------

1. Overall Operation
sed by default copies standard input to standard output, possibly performing one or more editing commands on each line before writing it to output. This behavior can be changed with flags on the command line; see section 1.1 below.

The general format of an editing command is:


One or two addresses may be omitted; the format of addresses is given in section 2. Any number of blanks or tabs can separate the address and the function. The function must be present; all available commands are discussed in section 3. Depending on which function is given, parameters may be required or optional; they are discussed under each individual function in section 3.

Ignore tab characters and spaces at the start of these lines.

1.1. Command-line Flags
Three flags are recognized on the command line:

-n: Tells sed not to copy all lines, only lines specified by the p function or the p flag after the s function (see section 3.3).
-e: Tells sed to accept the next parameter as an editing command.
-f: Tells sed to accept the next parameter as a filename; this file should contain one editing command per line.
1.2. Order of Application of Editing Commands
All editing commands are compiled into an efficient form for execution before any editing is done (in fact, even before any file is opened). These commands are compiled in the order they appear; generally, this is also the order in which they are attempted during execution. These commands are applied one at a time; the input to each command is the output of all previous commands.

The default linear order of command application can be changed with control flow commands t and b (see section 3). Even when the application order is changed by these commands, the input to any command is still the output of any previously applied commands.

1.3. Pattern Space
The range of pattern matching is called the pattern space. Generally, the pattern space is a line of input text, but more than one line can be read into the pattern space by using the N command (see section 3.6.).

1.4. Examples
Examples are scattered throughout the text. Unless otherwise specified, examples assume the following input text:

In Xanadu did Kubla Khan
A stately pleasure dome decree:
Where Alph, the sacred river, ran
Through caverns measureless to man
Down to a sunless sea.
(In no case should the output of sed commands be considered an improvement on Coleridge's work.)

Examples:
Command

2q
will exit after copying the first two lines of input. The output will be:

In Xanadu did Kubla Khan
A stately pleasure dome decree:

--------------------------------------------------------------------------------

2. Addresses: Selecting Lines to Edit
Lines in the input file to which editing commands are applied can be selected by addresses. Addresses can be line numbers or context addresses.

By grouping commands with braces ('{ }'), a single address (or address pair) can control the application of a group of commands (see section 3.6.).

2.1. Line Number Addresses
Line numbers are decimal integers. As each line is read from input, a line number counter is incremented; a line number address matches (selects) the input line whose internal counter equals the address line number. The counter runs cumulatively across multiple input files and is not reset when a new file is opened.

As a special case, the character $ matches the last line of the input file.

2.2. Context Addresses
Context addresses are patterns enclosed in slashes ('/'). The regular expressions recognized by sed are constructed as follows:

1) A normal character (not one of the characters discussed below) is a regular expression and matches that character.
2) The '^' symbol at the start of a regular expression matches the null character at the start of the line.
3) The dollar sign '$' at the end of a regular expression matches the null character at the end of the line.
4) The character '\n' matches an embedded newline character, not the terminating newline of the pattern space.
5) The dot '.' matches any character except the terminating newline of the pattern space.
6) A regular expression followed by an asterisk '*' matches any number (including 0) of contiguous occurrences of the regular expression it follows.
7) A string inside square brackets '' matches any character in the string, and no others. But if the first character of the string is the '^' symbol, the regular expression matches any character except those in the string and the terminating newline of the pattern space.
8) The concatenation of regular expressions is a regular expression that matches the concatenation of the strings matched by the members of the regular expression.
9) A regular expression between sequential '\(' and '\)' has the same effect as the regular expression without it, but it has a side effect described in the s command below and in rule 10 immediately following.
10) The expression '\d' means the same string as the expression enclosed by the previous '\(' and '\)' in the same expression. Here, d is a single digit; the specified string is the string starting at the d-th occurrence of '\(' from the left. For example, the expression '^\(.*\)\1' matches lines that start with two repeated occurrences of the same string.
11) An isolated empty regular expression (i.e., '//') is equivalent to the last compiled regular expression.
To use one of the special characters (^ $ . * \ /) as a literal (to match their own occurrence in the input), precede the special character with a backslash '\'.

A context address 'matches' the input if the entire pattern in the address matches part of the pattern space.

2.3. Number of Addresses
Commands in the next section may have 0, 1, or 2 addresses. The maximum number of addresses allowed is given for each command. Commands with more than the maximum number of addresses are considered errors.

If a command has no addresses, it is applied to every line in the input.

If a command has one address, it is applied to all lines that match that address.

If a command has two addresses, it is applied to the first line that matches the first address, and all subsequent lines up to and including the first subsequent line that matches the second address. Then the first address is again attempted on subsequent lines, and this process repeats.

Two addresses are separated by a comma.

Examples:
/an/ matches lines 1, 3, 4 of our sample text
/an.*an/ matches line 1
/^an/ no matching lines
/./ matches all lines
/\./ matches line 5
/r*an/ matches lines 1, 3, 4 (number = zero!)
/\(an\).*\1/ matches line 1

3. Functions
All functions are named by a single character. In the following summary, the maximum number of addresses allowed is given in parentheses, followed by the single character function name, possible parameters enclosed in angle brackets (< >), an English explanation of the single character name, and finally a description of what each function does. The angle brackets around parameters are not part of the parameters and should not be typed in actual editing commands.

3.1. Line-oriented Functions
(2)d -- delete lines
The d function deletes (does not write to output) all lines that match its address.

It also has the side effect that no further commands are attempted on the deleted line; after executing d, a new line is read from input and the editing command list is restarted from the beginning on the new line.

(2)n -- next line
The n function reads the next line from input, replacing the current line. The current line is written to output if appropriate. Execution continues with the part of the editing command list after the n command.

(1)a\
<text> -- append lines
The a function causes the parameter <text> to be written to output after the line that matches its address. The a command is inherently multi-line; a must appear at the end of a line, and <text> can contain any number of lines. To maintain the concept of one command per line, internal newlines must be hidden by preceding them immediately with a backslash character ('\'). The <text> parameter ends at the first unhidden newline (the first newline not immediately preceded by a backslash).

Once the a function is successfully executed, <text> will be written to output regardless of what later commands do to the triggering line. The triggering line can be completely deleted; <text> will still be written to output.

<text> is not scanned by the address match and no editing commands are attempted on it. It does not cause any change to the line number counter.

(1)i\
<text> -- insert lines
The i function behaves identically to the a function, except that <text> is written to output before the matching line. All other comments about the a function apply equally to the i function.

(2)c\
<text> -- change lines
The c function deletes the lines selected by its address and replaces them with the lines in <text>. Like a and i, c must be followed by newlines hidden by backslashes; and internal newlines in <text> must be hidden by backslashes.

The c command can have two addresses, so it can select a range of lines. If found, all lines in this range are deleted, and only one copy of <text> is written to output, not one copy for each deleted line. Like a and i, <text> is not scanned by the address match and no editing commands are attempted on it. It does not cause any change to the line number counter.

Once a line has been deleted by the c function, no further commands are attempted on the deleted line.

If the a or r function appends text after a line, and this line is later changed by the c function, the text inserted by the c function will be placed before the text of the a or r function. (The r function is described in section 3.4.).

Note: Leading blanks and tabs in the text placed by these functions disappear, just like in sed editing commands. To put leading blanks and tabs in output, precede the first desired blank or tab with a backslash; this backslash will not appear in the output.

Examples:
The list of editing commands:

n
a\
XXXX
d
applied to our standard input generates:

In Xanadu did Kubhla Khan
XXXX
Where Alph, the sacred river, ran
XXXX
Down to a sunless sea.
In this particular case, the following two lists of commands will generate the same effect:

n n
i\ c\
XXXX XXXX
d

3.2. Substitution Functions
This is a very important function that changes part of a line selected by a context search.

(2)s<pattern><replacement><flags> -- substitute
The s function substitutes part of the line (selected by <pattern>) with <replacement>. It can be read as:

Substitute <pattern> with <replacement>

The <pattern> parameter contains a pattern that is exactly the same as the pattern in an address (see section 2.2). The only difference between a context address and <pattern> is that a context address must be bounded by slash characters ('/'); <pattern> can be bounded by any other character that is not a space or newline.

By default, only the first occurrence of <pattern> is substituted; see the g flag below.

The <replacement> parameter starts immediately after the second delimiter character of <pattern> and must be immediately followed by another instance of the delimiter character. (So there are exactly three instances of the delimiter character). <replacement> is not a pattern, and characters with special meaning in the pattern have no special meaning in <replacement>. Instead, the special characters are:

& is replaced with the string that matched <pattern>.
\d (where d is a single digit) is replaced with the substring that matched the d-th part enclosed in '\(' and '\)' in <pattern>. If there are nested substrings in <pattern>, the d-th is counted by opening delimiters ('\('). As in the pattern, special characters can be made literal by preceding them with a backslash ('\').
The <flags> parameter can contain any of the following flags:

g -- substitute all (non-overlapping) instances of <pattern> in this line with <replacement>, starting the scan for the next instance of <pattern> after the inserted characters; characters from <replacement> placed in the line are not re-scanned.
p -- print this line if a successful substitution is made. The p flag causes the input line to be written to output if and only if this s function actually made a substitution. Note that if there are multiple s functions each followed by a p flag, and they all successfully make substitutions on the same input line, multiple copies of this line will be written to output: one copy for each successful substitution.
w <filename> -- write this line to a file if a successful substitution is made. The w flag causes the lines actually substituted by the s function to be written to the file named by <filename>. If <filename> exists before sed runs, it is overwritten. Otherwise, it is created.
w and <filename> must be separated by a single space.

There is the possibility of writing multiple slightly different copies of an input line, just like with the p flag.

The maximum number of different file names that can be referred to after the w flag and the w function (see section below) is 10.

Examples:
Applying the following command to our standard input,

s/to/by/w changes
produces, on standard output:

In Xanadu did Kubhla Khan
A stately pleasure dome decree:
Where Alph, the sacred river, ran
Through caverns measureless by man
Down by a sunless sea.
In file 'changes':

Through caverns measureless by man
Down by a sunless sea.
If the no-copy option is in effect, the command:

s//*P&*/gp
produces:

A stately pleasure dome decree*P:*
Where Alph*P,* the sacred river*P,* ran
Down to a sunless sea*P.*

Finally, to show the effect of the g flag, the command:

/X/s/an/AN/p
produces (assuming no copy mode):

In XANadu did Kubhla Khan

and the command:

/X/s/an/AN/gp
produces:

In XANadu did Kubhla KhAN
3.3. Input/Output Functions
(2)p -- print
The print function writes the addressed line to standard output. It is written when the p function is encountered, regardless of what subsequent editing commands do to these lines.

(2)w <filename> -- write on <filename>
The write function writes the addressed line to the file named by <filename>. If the file already existed before, it is overwritten; otherwise, it is created. Each line is written as it exists when the write function is encountered, regardless of what subsequent editing commands do to these lines. w and <filename> must be separated by exactly one space. The maximum number of different file names that can be referred to after the w flag in the s function and in the write function is 10.

(1)r <filename> -- read the contents of a file
The read function reads the contents of <filename> and appends them after the line that matches this address. The file is read and its contents are appended, regardless of what subsequent editing commands do to the lines that match its address. If the r and a functions are executed on the same line, the text from the a function and the r function is written to output in the order the functions are executed. r and <filename> must be separated by exactly one space. If the file referred to by the r function cannot be opened, it is treated as an empty file, not an error, so no diagnostic message is given.

Note: Because there is a limit on the number of files that can be opened simultaneously, be careful not to refer to more than 10 (different) files in the w command or flag; if any r function is present, this number is reduced by one. (Only one read file can be open at a time).

Examples
Assume file 'note1' has the following content:

Note: Kubla Khan (more properly Kublai Khan; 1216-1294) was the grandson and most eminent successor of Genghiz (Chingiz) Khan, and founder of the Mongol dynasty in China.
Then the following command:

/Kubla/r note1
produces:

In Xanadu did Kubla Khan
Note: Kubla Khan (more properly Kublai Khan; 1216-1294) was the grandson and most eminent successor of Genghiz (Chingiz) Khan, and founder of the Mongol dynasty in China.
A stately pleasure dome decree:
Where Alph, the sacred river, ran
Through caverns measureless to man
Down to a sunless sea.

3.4. Multi-line Input Functions
There are three functions spelled in uppercase that handle pattern spaces containing embedded newlines; they are mainly intended to provide pattern matching across lines in input.

(2)N -- Next line
Adds the next line to the current line in the pattern space; the two input lines are separated by an embedded newline. Pattern matching can extend across this embedded newline.

(2)D -- Delete first part of the pattern space
Deletes all characters in the current pattern space up to and including the first newline character. If the pattern space becomes empty (the only newline is the terminating newline), another line is read from input. In any case, execution is restarted from the beginning of the editing command list.

(2)P -- Print first part of the pattern space
Prints all characters in the pattern space up to and including the first newline.

The P and D functions are equivalent to their corresponding lowercase functions if there is no embedded newline in the pattern space.

3.5. Save and Retrieve Functions
There are four functions that save and retrieve parts of input for future use.

(2)h -- hold pattern space
The h function copies the contents of the pattern space to the hold area (destroying the previous contents of the hold area).

(2)H -- Hold pattern space
The H function appends the contents of the pattern space to the contents of the hold area; the previous and new contents are separated by a newline.

(2)g -- get contents of hold area
The g function copies the contents of the hold area to the pattern space (destroying the previous contents of the pattern space).

(2)G -- Get contents of hold area
The G function appends the contents of the hold area to the contents of the pattern space; the previous and new contents are separated by a newline.

(2)x -- exchange
The exchange command swaps the contents of the pattern space and the hold area.

Examples
Command

1h
1s/ did.*//
1x
G
s/\n/ :/
applied to our standard example produces:

In Xanadu did Kubla Khan :In Xanadu
A stately pleasure dome decree: :In Xanadu
Where Alph, the sacred river, ran :In Xanadu
Through caverns measureless to man :In Xanadu
Down to a sunless sea. :In Xanadu

3.6. Control Flow Functions
These functions do not edit lines in input but control the application of functions to lines selected by the address part.

(2)! -- Don’t
The not command causes the next command written on the same line to be applied to all and only the input lines not selected by the address part.

(2){ -- Grouping
The grouping command '{' causes the next group of commands to be applied (or not applied) as a block to the input lines selected by the address of the grouping command. The first command in the grouped control can appear on the same line as '{' or the next line.

The grouped commands are terminated by a matching '}' on its own line.

Grouping can be nested.

(0):<label> -- place a label
The label function marks a position in the editing command list that can be referred to later by the b and t functions. <label> can be any sequence of eight or fewer characters; if two different colon functions have the same label, a compile-time diagnostic message is generated and no execution attempt is made.

(2)b<label> -- branch to label
The branch function causes the sequence of editing commands applied to the current input line to be restarted immediately after the colon function with the same <label>. If a colon function with the same label is not found after all editing commands have been compiled, a compile-time diagnostic message is generated and no execution attempt is made.

A b function without <label> is treated as a branch to the end of the editing command list; whatever processing is to be done on the current input line is done, and other input lines are read; the editing command list is restarted from the beginning on this new line.

(2)t<label> -- test substitutions
The t function tests whether any successful substitutions have been made on the current input line; if so, it branches to <label>; otherwise, it does nothing. The flag indicating that a successful substitution has been executed is reset by:

1) Reading a new input line, or
2) Executing a and t functions.
3.7. Miscellaneous Functions
(1)= -- equals
The = function writes the line number of the line that matches its address to standard output.

(1)q -- quit
The q function causes the current line to be written to standard output (if appropriate), any added or read text to be written out, and execution to be terminated.
--------------------------------------------------------------------------------
References
Ken Thompson and Dennis M. Ritchie, The UNIX Programmer’s Manual. Bell Laboratories, 1978.

Last edited by 无奈何 on 2006-10-26 at 01:33 PM ]

Repost Note: This is a chapter from *sed & awk*, translated by a netizen, original address unknown.
Advanced sed Usage <Collection>

First of all, you should understand the definition of the pattern space. The pattern space is the cache where the line is read. All processing of the text line by sed is carried out in this cache. This is helpful for the next study.
In normal circumstances, sed reads the line to be processed into the pattern space, and the commands in the script are executed one after another on this line until the script is executed, then the line is output and the pattern space is cleared; then the above action is repeated, and the new line in the file is read until the file processing is complete.
However, for various reasons, such as the user wanting a certain command in the script to be executed under a certain condition, or wanting the pattern space to be retained for the next processing, it may cause sed not to follow the normal process when processing the file. At this time, sed sets some advanced commands to meet the user's requirements.
In general, these commands can be divided into the following three categories:
1. N, D, P: Deal with the problem of multi-line pattern space;
2. H, h, G, g, x: Put the content of the pattern space into the storage space for subsequent editing;
3. :, b, t: Implement branch and conditional structures in the script.
Processing of multi-line pattern space:
Since regular expressions are line-oriented, if a phrase is part at the end of one line and part at the beginning of the next line, it is quite difficult to handle with commands like grep. However, with the multi-line commands N, D, P of sed, this task can be easily completed.
The multi-line Next (N) command is relative to the next (n) command. The latter outputs the content in the pattern space, then reads the next line into the pattern space, but the script does not transfer to the beginning but starts executing from the current n command; while the former saves the original content in the pattern space, then reads the new line, and the two are separated by a newline character "\n". After the N command is executed, the control flow will continue to process the pattern space with the commands after the N command.
It is worth noting that in the multi-line mode, the special characters "^" and "$" match the very beginning and the very end of the pattern space, not the beginning and end of the embedded "\n".
Example 1:
$ cat expl.1
Consult Section 3.1 in the Owner and Operator
Guide for a description of the tape drives
available on your system.
Now we want to replace "Owner and Operator Guide" with "Installation Guide":
$ sed '/Operator$/{
> N
> s/Owner and Operator\nGuide/Installation Guide\
> /
> }' expl.1
In the above example, note that there is an embedded newline character between lines; also, if you want to insert a newline character in the replacement content, you need to use the escape of "\" as above.
Let's look at another example:
Example 2:
$ cat expl.2
Consult Section 3.1 in the Owner and Operator
Guide for a description of the tape drives
available on your system.

Look in the Owner and Operator Guide shipped with your system.

Two manuals are provided including the Owner and
Operator Guide and the User Guide.

The Owner and Operator Guide is shipped with your system.
$ sed 's/Owner and Operator Guide/Installation Guide/
> /Owner/{
> N
> s/ *\n/ /
> s/Owner and Operator Guide */Installation Guide\
> /
}' expl.2
The result is:
Consult Section 3.1 in the Installation Guide
for a description of the tape drives
available on your system.

Look in the Installation Guide shipped with your system.

Two manuals are provided including the Installation Guide
and the User Guide.

The Installation Guide is shipped with your system.
It seems that the two replacements in the sed command are redundant. In fact, if the first replacement is removed and the script is run again, it will be found that there are two problems in the output. One is that the last line in the result will not be replaced (in some versions of sed, it will not even be output). This is because the last line matches "Owner", the N command is executed, but it has reached the end of the file, some versions will directly print this line and exit, while others will not print and exit immediately. This problem can be solved by the command "$!N", which means that the N command has no effect on the last line. Another problem is that the "look manuals" paragraph is split into two lines, and the blank line with the next paragraph is deleted. This is the result of the embedded newline character being replaced. Therefore, the two replacements in sed are not redundant at all.
Example 3:
$ cat expl.3
<para>

This is a test paragraph in Interleaf style ASCII. Another line
in a paragraph. Yet another.

<Figure Begin>

v.1111111111111111111111100000000000000000001111111111111000000
100001000100100010001000001000000000000000000000000000000000000
000000

<Figure End>

<para>

More lines of text to be found after the figure.
These lines should print.
Our sed command is as follows:
$ sed '/<para>{
> N
> c\
> .LP
> }
> /<Figure Begin>/,/<Figure End>/{
> w fig.interleaf
> /<Figure End>/i\
> .FG\
> <insert figure here>\
> .FE
> d
> }
> /^$/d' expl.3
After running, the result is:
.LP
This is a test paragraph in Interleaf style ASCII. Another line
in a paragraph. Yet another.
.FG
<insert figure here>
.FE
.LP
More lines of text to e found after the figure.
These lines should print.
And the content between <Figure Begin> and <Figure End> is written to the file "fig.interleaf". It is worth noting that the command "d" will not affect the content inserted by the command i.
The function of the command "d" is to delete the content in the pattern space, then read a new line, and the sed script starts executing again from the beginning. The difference from the command "D" is that it deletes part of the pattern space up to the first embedded newline character, but does not read a new line, and the script will return to the beginning to process the remaining content.
Example 4:
$ cat expl.4
This line is followed by 1 blank line.

This line is followed by 2 blank line.


This line is followed by 3 blank line.



This line is followed by 4 blank line.




This is the end.
Different delete commands get different results:
$ sed '/^$/{ $ sed '/^$/{
> N > N
> /^\n$/d > /^\n$/D
> }' expl.4 > }' expl.4
The default action of sed for each line in the file (regardless of whether it is processed or not) is to output it. If the option "-n" is added, the output action will be suppressed. At this time, if you still want to output, you need the print command. The print command for the single-line pattern space is "p", and the print command for the multi-line pattern space is "P". The P command prints part of the pattern space up to the first embedded newline character.
The P command usually appears after the N command and before the D command, thus forming an input-output loop. In this case, there are always two lines of text in the pattern space, and the output is always one line of text. The purpose of using this loop is to output the first line in the pattern space, then the script returns to the beginning, and then processes the second line in the space. Imagine that if there is no this loop, when the script is executed completely, the content in the pattern space will all be output, which may not meet the user's requirements or reduce the efficiency of the program execution.
Here is an example:
Example 5:
$ cat expl.5
Here are examples of the UNIX
System. Where UNIX
System appears, it should be the UNIX
Operating System.
$ sed '/UNIX$/{
> N
> /\nSystem/{
> s// Operating &/
> P
> D
> }
> }' expl.5
The result of the replacement is:
Here are examples of the UNIX Operating
System. Where UNIX Operating
System appears, it should be the UNIX
Operating System.
You can replace "P" and "D" in the sed command with lowercase letters to compare the differences between the two types of commands.
The following example is quite difficult:
Example 6:
$ cat expl.6
I want to see @fl(what will happen) if we put the
font change commands @fl(on a set of lines). If I understand
things (correctly), the @fl(third) line causes problems. (No?).
Is this really the case, or is it (maybe) just something else?

Let's test having two on a line @fl(here) and @fl(there) as
well as one that begins on one line and ends @fl(somewhere
on another line). What if @fl(it is here) on the line?
Another @fl(one).
Now what needs to be done is to replace "@fl(…)" with "\fB(…)\fR. The following is the sed command that meets the conditions:
$ sed 's/@fl(\(*\))/\\fB\1\\fR/g
> /@fl(.*/{
> N
> s/@fl(\(.*\n*\))/\\fB\1\\fR/g
> P
> D
> }' expl.6
However, if this input-output loop is not used and only N is used to implement it, problems will occur:
$ sed 's/@fl(\(*\))/\\fB\1\\fR/g
> /@fl(.*/{
> N
> s/@fl(\(.*\n*\))/\\fB\1\\fR/g
> }' expl.6
This sed script has loopholes.


Storing lines:
The definition of the pattern space has been explained earlier, and there is another cache called the storage space in sed. The content in the pattern space and the storage space can be copied to each other through a set of commands:
Command Abbreviation Function
Hold h or H Copy or append the content of the pattern space to the storage space
Get g or G Copy or append the content of the storage space to the pattern space
Exchange x Exchange the content in the pattern space and the storage space
The difference between uppercase and lowercase of the commands is that the uppercase command appends the content of the source space to the target space, while the lowercase command overwrites the target space with the content of the source space. It is worth noting that whether it is the Hold command or the Get command, a newline character will be added after the original content of the destination space, and then the content of the source space will be added after the newline character.
From the following example, you can experience the initial application of this part of the content:
Example 7:
$ cat expl.7
1
2
11
22
111
222
The work we need to do is to swap the first line with the second line, the third line with the fourth line, and the fifth line with the sixth line. The sed command format is:
$ sed '
> /1/{
> h
> d
> }
> /2/{
> G
> }' expl.7
The process is as follows: First, sed reads the first line into the pattern space, then the h command puts it into the storage space to save it, and a d command clears the content in the pattern space; then sed reads the second line into the pattern space, and then the G command appends the content in the storage space to the pattern space (note that a newline character is added at the end of the original content in the pattern space).
The final result is as follows:
2
1
22
11
222
111
When using the H or h command, it is more common to add the d command after this command. In this way, the sed script will not reach the end, and thus the content in the pattern space will not be output. In addition, if d is replaced with n, or G is replaced with g, the purpose will not be achieved.
What is the most convenient way to convert the case of letters? Probably tr.
$ tr "" "" File
It is very powerful that sed can also complete this conversion. The corresponding command is y:
$ sed '
> //y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' File
However, the y command completely modifies the entire line, so if you only want to convert the case of a few characters in the line, this will not work. To complete this work, you need to use the Hold and Get commands mentioned above.
cat expl.8
find the Match statement
Consult the Get statement
using the Read statement to retrieve data
$ sed '/the .* statement/{
> h
> s/.*the \(.*\) statement.*/\1/
> y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/
> G
> s/\(.*\)\n\(.*the \).*\( statement.*\)/\2\1\3/
> }' expl.8
Take the processing process of the first line to illustrate the meaning of this command:
(1) "find the Match statement" is put into the storage space;
(2) Replace this line to get: Match;
(3) Convert the result of (2) to uppercase: MATCH;
(4) Take out the content reserved in (1) from the storage space and append it to the pattern space. At this time, the content in the pattern space is:
MATCH\nfind the Match statement
(5) Replace the content in the pattern space again to get: find the MATCH statement.
The following example will use relatively solid regular expressions, but it doesn't matter, take your time, and all problems can be solved. Also, the text used in this example is mainly related to editing and typesetting. I am not very good at this, so I will just take out the sed script, grasp the core, and omit the trivial details:
Example 9:
$ cat expl.9.sed
h
s//\\&/g
x
s//\\&/g
s/^\.XX //
s/$/\//
x
s/^\\\.XX \(.*\)$/\/^\\.XX \/s\/\1//
G
s/\n//
(1) h: Put the text line into the storage space.
(2) s//\\&/g: This expression is relatively difficult. If the first character in the class expression, that is, "]", then "]" loses its special meaning; in addition, in "", only "\" has a special meaning, which means that "*" and "." are all understood as literal meanings. To make them have special meanings, it is necessary to use the escape of "\"; although it does not appear in the expression, it should be mentioned that in "", only "^" at the first position means "not", and the rest are literal explanations, and "$" only has a special meaning at the end of the regular expression. "\\" removes the special meaning of "\" and "&" represents the forward reference. Therefore, the meaning of the second command is: replace " "、"\"、"*"、"." in the pattern space with "\"、"\\"、"\*"、"\." in turn.
(3) x: Exchange the pattern space and the storage space. After executing this command, the content in the pattern space is the original content, and the content in the storage space changes, and each special character is replaced with "\&".
(4) s//\\&/g: Process the pattern space, and any "\" or "&" that appear will be replaced with "\\" or "\&".
(5) s/$/\//: This is easy to understand, that is, add a "/" at the end of the pattern space.
(6) x: Exchange the contents of the two spaces again.
(7) s/^\\\.XX \(.*\)$/\/^\\.XX \/s\/\1//: There is no difficulty in this, it is just that a few references are easy to confuse people. Be careful and there will be no problem, so I will skip it.
(8) G: Skipped.
(9) s/\n//: Delete the newline character.
What is the use of this script? It will be clear when experimenting with the following text:
.XX "asterisk (*) metacharacter"
The following are the results of each command, the first line and the second line respectively represent the content of the pattern space and the storage space:
1. .XX "asterisk (*) metacharacter"
.XX "asterisk (*) metacharacter"

2. \.XX "asterisk (\*) metacharacter"
.XX "asterisk (*) metacharacter"

3. .XX "asterisk (*) metacharacter"
\.XX "asterisk (\*) metacharacter"

4. .XX "asterisk (*) metacharacter"
\.XX "asterisk (\*) metacharacter"

5. "asterisk (*) metacharacter"
\.XX "asterisk (\*) metacharacter"

6. "asterisk (*) metacharacter"/
\.XX "asterisk (\*) metacharacter"

7. \.XX "asterisk (\*) metacharacter"
"asterisk (*) metacharacter"/

8. /^\.XX /s/"asterisk (\*) metacharacter"/
"asterisk (*) metacharacter"/

9. /^\.XX /s/"asterisk (\*) metacharacter"/\n/"asterisk (*) metacharacter"/

10./^\.XX /s/"asterisk (\*) metacharacter"/"asterisk (*) metacharacter"/
You see, in fact, "s//\\&/" did not work in our example, but it is indispensable because in the second part of the s command, "\" and "&" have special meanings, so their special meanings need to be escaped in advance.
Understood? When you want to use a shell script to automatically generate a sed script mainly for replacement commands, you will find how crucial the above content is for the processing of special characters.
In addition to the above applications, the storage space can even store the content of many lines for subsequent output. In fact, this function is very effective for texts with very obvious structures such as html. The following is a related example:
Example 10
cat expl.10
<p>My wife won't let me buy a power saw. She is afraid of an
accident if I use one.
So I rely on a hand saw for a variety of weekend projects like
building shelves.
However, if I made my living as a carpenter, I would
have to use a power
saw. The speed and efficiency provided by power tools
would be essential to being productive.</p>

<p>For people who create and modify text files,
sed and awk are power tools for editing.</p>

<p>Most of the things that you can do with these programs
can be done interactively with a text editor. However,
using these programs can save many hours of repetitive
work in achieving the same result.</p>

$ sed '/^$/!{
> H
> d
> }
> /^$/{
> x
> s/^\n/<p>/
> s/$/<\/p>/
> G
> }' expl.10
Run this command and see what the result is. In fact, the result is not important. Through this child, what should be learned is the thought of process control reflected in the script. The first part of the script uses "!" to indicate processing for lines that do not match, but this processing will not go to the bottom of the script because of the existence of "d", so there will be no output; in the second part of the script, the script does reach the end, and the content of the pattern space and the storage space is cleared accordingly, preparing for reading the next paragraph.
Originally, this example is over, but there is another situation. What will happen if the last line of the file is not an empty line? Obviously, the last paragraph of the text will not be output. How to handle this situation? The wisest way is to "manufacture" an empty line by yourself. The new script is as follows:
$ sed '${
> /^$/!{
> H
> s/.*//
> }
> }
> /^$/!{
> H
> d
> }
> /^$/{
> x
> s/^\n/<p>/
> s/$/<\/p>/
> G
> }' expl.10


Flow control commands
In order to make users really "free" when writing sed scripts, sed also allows setting markers with ":" in the script, and then using "b" and "t" commands for process control. As the name implies, "b" means "branch" and "t" means "test"; the former is the branch command, and the latter is the test command.
First, let's see what the format of the label is. This label is placed where you want the process to start, on a separate line, starting with a colon. There should be no spaces or tabs between the colon and the transition, and if there are spaces at the end of the label, they will also be considered as part of the label.
Let's talk about the b command. Its format is as follows:
b
Its meaning is that if the address is satisfied, the sed process follows the label to jump: if the label is specified, the script first assumes that this label is on a certain line below the b command, and then transfers to that line to execute the corresponding command; if this label does not exist, the control flow directly jumps to the end of the script. Otherwise, continue to execute the subsequent commands.
In some cases, the b command is somewhat similar to the! command, but the! command only works on the content in the {} immediately next to it, while the b command gives the user enough freedom to choose which commands should be executed and which should not be executed in the sed script. The following are several classic usages of the b command:
(1) Create a loop:
:top
command1
command2
/pattern/b top
command3
(2) Ignore some commands that do not meet the conditions:
command1
/patern/b end
command2
:end
command3
(3) Only one of the two parts of the command can be executed:
command1
/pattern/b dothere
command
b
:dothere
command3
The format of the t command is the same as that of the b command:
t
It means that if the address is satisfied, the sed script will transfer the process according to the label indicated by the t command. The rules for the label are the same as those for the b command mentioned above. Here is also an example:
s/pattern/replacement/
t break
command
:break
Let's take the sed script of Example 6 as an example. In fact, if you think carefully, you will find that this script is not powerful enough: what if a @fl structure spans three lines? This requires the following enhanced version of sed:
$ cat expl.6.sed
:begin
/@fl(\(*\))/{
s//\\fB\1\\fR/g
b begin
}
/@fl(.*/{
N
s/@f1(\(*\n*\))/\\fB\1\\fR/g
t again
b begin
}
:again
P
D

Last edited by 无奈何 on 2006-10-26 at 01:42 PM ]

Repost Note: Original link http://www-128.ibm.com/developerworks/cn/linux/shell/sed/sed-1/index.html
General Thread -- sed Examples, Part 1
Daniel Robbins, President/CEO, Gentoo Technologies, Inc.

October 2001

In this article series, Daniel Robbins will show you how to use the very powerful (but often forgotten) UNIX stream editor sed. sed is an ideal tool for editing files in batch mode or creating shell scripts in a very efficient way to modify existing files.
Choosing an Editor
There are many text editors available in the UNIX world for us to choose from. Think about it -- vi, emacs, jed, and many other tools will come to mind. We all have editors that we have gradually come to know and love (and our favorite key combinations). With a reliable editor, we can easily handle any number of UNIX-related administrative or programming tasks.

Although interactive editors are great, they have their limitations. Although their interactive nature can be a strength, it also has its drawbacks. Consider a situation where you need to make similar changes to a set of files. You might instinctively run your favorite editor and then manually perform a set of tedious, repetitive, and time-consuming editing tasks. However, there is a better way.
Enter sed
It would be great if we could automate the process of editing files so that we can edit files in "batch" mode or even write scripts that can make complex changes to existing files. Fortunately, for this situation, there is a better way -- this better way is called "sed".

sed is a lightweight stream editor included in almost all UNIX platforms, including Linux. sed has many good features. First of all, it is quite small, usually much smaller than your favorite scripting language. Second, because sed is a stream editor, it can edit data received from standard input such as pipes. Therefore, there is no need to store the data to be edited in a file on disk. Since data can be easily piped to sed, it is easy to use sed as a long and complex pipe in a powerful shell script. Try doing that with your favorite editor.

GNU sed
Fortunately for Linux users, one of the best versions of sed is exactly GNU sed, whose current version is 3.02. Every Linux distribution has (or at least should have) GNU sed. GNU sed is popular not only because its source code can be freely distributed, but also because it happens to have many convenient and time-saving extensions to the POSIX sed standard. In addition, GNU does not have many of the limitations of the early specialized versions of sed, such as line length limits -- GNU can easily handle lines of any length.
Latest GNU sed
When I was researching this article, I noticed that several online sed enthusiasts mentioned GNU sed 3.02a. Strangely, sed 3.02a cannot be found on ftp.gnu.org (see Resources for these links), so I had to look elsewhere. I found it in /pub/sed on alpha.gnu.org. So I happily downloaded it, compiled it, and installed it, and a few minutes later I found that the latest version of sed was 3.02.80 -- its source code can be found next to the 3.02a source code on alpha.gnu.org. After installing GNU sed 3.02.80, I was completely ready.

alpha.gnu.org (see Resources) is the home of new and experimental GNU source code. However, you will also find many excellent, stable source code there. For some reason, either many GNU developers have forgotten to move the stable source code to ftp.gnu.org, or their "beta" period is exceptionally long (2 years!). For example, sed 3.02a has been around for two years, and even 3.02.80 has been around for a year, but they are still not available on ftp.gnu.org (when this article was written in August 2000).
Proper sed
In this series, we will use GNU sed 3.02.80. In the upcoming subsequent articles in this series, some (but very few) of the most advanced examples will not work in GNU sed 3.02 or 3.02a. If you are not using GNU sed, the results may be different. Why not take some time now to install GNU sed 3.02.80? That way, you will not only be ready for the rest of this series, but you will also be using what may be the best sed available at present.
sed Examples
sed works by performing any number of user-specified editing operations ("commands") on the input data. sed is line-based, so it executes commands on each line in sequence. sed then writes its results to standard output (stdout), and it does not modify any input files.

Let's look at some examples. The first few will be a bit strange because I will use them to demonstrate how sed works, not to perform any useful tasks. However, if you are new to sed, it is very important to understand them. Here is the first example:


$ sed -e 'd' /etc/services



If you enter this command, you will get no output. So, what happened? In this example, sed is called with an editing command 'd'. sed opens the /etc/services file, reads a line into its pattern buffer, executes the edit command ("delete the line"), and then prints the pattern buffer (the buffer is empty). Then, it repeats these steps for each subsequent line. This produces no output because the "d" command removes each line in the pattern buffer!

There are a few other things to notice in this example. First, /etc/services is not modified at all. This is still because sed only reads the file specified on the command line and uses it as input -- it does not attempt to modify the file. The second thing to notice is that sed is line-based. The 'd' command does not simply tell sed to delete all input data at once. Instead, sed reads each line of /etc/services into its internal buffer called the pattern buffer line by line. Once a line is read into the pattern buffer, it executes the 'd' command and then prints the contents of the pattern buffer (there is nothing in this example). I will show you later how to use address ranges to control which lines the commands are applied to -- but if no address is used, the command will be applied to all lines.

The third thing to notice is the use of single quotes to enclose the 'd' command. It is a good idea to get into the habit of using single quotes to enclose sed commands so that shell expansion is disabled.
Another sed example
Here is an example of using sed to remove the first line of the /etc/services file from the output stream:


$ sed -e '1d' /etc/services | more



As you can see, this command is very similar to the first 'd' command except that there is a '1' in front. If you guessed that '1' refers to the first line, you are correct. Unlike the first example where only 'd' is used, this time a optional numeric address is used in front of 'd'. By using an address, you can tell sed to edit only certain specific lines.
Address Ranges
Now, let's see how to specify an address range. In this example, sed will delete lines 1 to 10 of the output:


$ sed -e '1,10d' /etc/services | more



When two addresses are separated by a comma, sed will apply the subsequent command to the range from the first address to the second address. In this example, the 'd' command is applied to lines 1 to 10 (inclusive). All other lines are ignored.
Addresses with Regular Expressions
Now let's demonstrate a more useful example. Suppose you want to view the contents of the /etc/services file, but you are not interested in viewing the comment sections included in it. As you know, comments can be placed in the /etc/services file by lines starting with the '#' character. To avoid comments, we want sed to delete lines that start with '#'. Here's how:


$ sed -e '/^#/d' /etc/services | more



Try this example and see what happens. You will notice that sed successfully accomplishes the intended task. Now, let's analyze what happened.

To understand the '/^#/d' command, first we need to parse it. First, let's remove 'd' -- this is the same delete line command we used earlier. The new addition is the '/^#/' part, which is a new regular expression address. Regular expression addresses are always enclosed in slashes. They specify a pattern, and the command that follows the regular expression address will only apply to lines that exactly match that particular pattern.

So, '/^#/' is a regular expression. But what does it do? Obviously, it's time to review regular expressions now.
Review of Regular Expressions
Regular expressions can be used to represent patterns that may be found in text. Have you used the '*' character on the shell command line? This usage is similar to regular expressions, but not the same. Here are the special characters that can be used in regular expressions:

Character Description
^ Matches the start of a line
$ Matches the end of a line
. Matches any single character
* Matches zero or more occurrences of the previous character
Matches any character within

Probably the best way to get a feel for regular expressions is to look at a few examples. All of these examples will be accepted by sed as valid addresses that appear on the left side of the command. Here are a few examples:

Regular Expression Description
/./ Matches any line that contains at least one character
/../ Matches any line that contains at least two characters
/^#/ Matches any line that starts with '#'
/^$/ Matches all empty lines
/}$/ Matches any line that ends with '}' (no spaces)
/} *$/ Matches any line that ends with '}' followed by zero or more spaces
// Matches any line that contains lowercase 'a', 'b', or 'c'
/^/ Matches any line that starts with 'a', 'b', or 'c'

In these examples, you are encouraged to try a few. Spend some time getting familiar with regular expressions, then try a few regular expressions you create yourself. You can use regexp as follows:


$ sed -e '/regexp/d' /path/to/my/test/file | more



This will cause sed to delete any matching lines. However, it is more helpful to get familiar with regular expressions by telling sed to print the regexp match and delete the non-matching content, rather than the other way around. You can do this with the following command:


$ sed -n -e '/regexp/p' /path/to/my/test/file | more



Note the new '-n' option, which tells sed not to print the pattern space unless explicitly asked to. You will also notice that we replaced the 'd' command with the 'p' command, which, as you might guess, explicitly asks sed to print the pattern space. That's it, only the matching part will be printed.
More on Addresses
So far, we have seen line addresses, line range addresses, and regexp addresses. But there are more possibilities. We can specify two regular expressions separated by a comma, and sed will match all lines from the first line that matches the first regular expression to the line that matches the second regular expression (inclusive). For example, the following command will print the text block starting from the line containing "BEGIN" and ending with the line containing "END":


$ sed -n -e '/BEGIN/,/END/p' /my/test/file | more



If "BEGIN" is not found, no data will be printed. If "BEGIN" is found but "END" is not found in all subsequent lines, all subsequent lines will be printed. This happens because of sed's stream-oriented nature -- it doesn't know if "END" will appear.
C Source Code Example
If you just want to print the main() function in a C source file, you can enter:


$ sed -n -e '/main]*(/,/^}/p' sourcefile.c | more



This command has two regular expressions '/main]*(/' and '/^}/', and a command 'p'. The first regular expression will match the string "main" followed by any number of spaces or tabs and then an opening parenthesis. This should match the start of a typical ANSI C main() declaration.

In this particular regular expression, the ']' character class appears. This is just a special keyword that tells sed to match TAB or space. If you want, you can type ']', then a space letter, then -V, then a tab letter and ']' -- Control-V tells bash to insert the "real" tab instead of performing command expansion. Using the ']' command class (especially in scripts) is clearer.

Okay, now let's look at the second regexp. '/^}' will match any '}' character that appears at the start of a new line. If the code is well-formatted, this will match the closing curly brace of the main() function. If the formatting is not good, it will not match correctly -- this is a tricky part of performing pattern matching tasks.

Because we are in '-n' quiet mode, the 'p' command still does its usual job, which is to explicitly tell sed to print the line. Try running this command on a C source file -- it should output the entire main() { } block, including the starting "main()" and the closing '}'.

General Thread -- sed Examples, Part 2


Content:

Substitution!

Regular Expression Mayhem

More Character Matching

Advanced Substitution Features

Those Wonderful Parentheses with Backslashes

Combining Commands

Multiple Commands for One Address

Appending, Inserting, and Changing Lines

Next Part

Resources

About the Author

Rating for this Article





October 2001

sed is a very powerful and compact text stream editor. In the second article of this series, Daniel Robbins shows you how to use sed to perform string substitution, create larger sed scripts, and how to use sed's appending, inserting, and changing line commands.
sed is a useful (but often forgotten) UNIX stream editor. It is an ideal tool for editing files in batch mode or creating shell scripts in an efficient way to modify existing files. This article is a continuation of the previous article introducing sed.

Substitution!
Let's take a look at one of the most useful commands in sed, the substitution command. With this command, you can replace a specific string or a matching regular expression with another string. Here is an example of the most basic usage of this command:


$ sed -e 's/foo/bar/' myfile.txt


The above command replaces the first occurrence (if any) of 'foo' in each line of myfile.txt with the string 'bar', and then outputs the contents of the file to standard output. Please note that I said the first occurrence per line, although this is usually not what you want. When performing string substitution, you usually want to perform a global substitution. That is, you want to replace all occurrences in each line, as follows:


$ sed -e 's/foo/bar/g' myfile.txt


The 'g' option appended after the last slash tells sed to perform a global substitution.

There are a few other things to know about the 's///' substitution command. First, it is a command, and it is just a command, and no address is specified in all the above examples. This means that 's///' can also be used with an address to control which lines the command is applied to, as follows:


$ sed -e '1,10s/enchantment/entrapment/g' myfile2.txt



The above example will cause all occurrences of the phrase 'enchantment' to be replaced with the phrase 'entrapment', but only on lines 1 to 10 (inclusive).


$ sed -e '/^$/,/^END/s/hills/mountains/g' myfile3.txt



This example will replace 'hills' with 'mountains', but only on the text block starting from an empty line and ending with a line starting with the three characters 'END' (inclusive).

Another nice thing about the 's///' command is that there are many substitution options for the '/' delimiter. If you are performing a string substitution and there are many slashes in the regular expression or substitution string, you can change the delimiter by specifying a different character after 's'. For example, the following example will replace all occurrences of /usr/local with /usr:


$ sed -e 's:/usr/local:/usr:g' mylist.txt


In this example, a colon is used as the delimiter. If you need to specify a delimiter character in the regular expression, you can precede it with a backslash.
Regular Expression Mayhem
So far, we have only performed simple string substitutions. Although this is convenient, we can also match regular expressions. For example, the following sed command will match a phrase starting with '<' and ending with '>', and containing any number of characters in between. The following example will delete this phrase (replace it with an empty string):


$ sed -e 's/<.*>//g' myfile.html


This is the first good attempt at a sed script to remove HTML tags from a file, but it won't work well because of the peculiar rules of regular expressions. Why? When sed tries to match a regular expression in a line, it looks for the longest match in the line. In my previous sed article, this was not a problem because we were using 'd' and 'p' commands, which always delete or print the entire line. However, when using the 's///' command, there is a big difference because the entire part matched by the regular expression will be replaced by the target string, or, in this example, deleted. This means that the following line:


<b>This</b> is what <b>I</b> meant.


will become:


meant.


We don't want this, but rather:


This is what I meant.


Fortunately, there is an easy way to correct this problem. Instead of entering a regular expression that is "< followed by some characters and ending with >", we just need to enter a regular expression that is "< followed by any number of non-> characters and ending with >". This will match the shortest, not the longest, possibility. The new command is as follows:


$ sed -e 's/<*>//g' myfile.html


In the above example, '' specifies "non->" characters, and the '*' after it completes the expression to mean "zero or more non->" characters. Test this command on a few html files, pipe the output to "more", and then carefully look at the results.
More Character Matching
There are some additional options for the '' regular expression syntax. To specify a range of characters, you can use '-' as long as the character is not in the first or last position, as follows:


'*'


This will match zero or more characters that are all 'a', 'b', 'c'...'v', 'w', 'x'. In addition, you can use the '' character class to match spaces. Here is a fairly complete list of available character classes:

Character Class Description
Alphanumeric
Alphabetic
Space or tab
Any control character
Digit
Any visible character (no space)
Lowercase
Non-control character
Punctuation character
Space
Uppercase
Hexadecimal digit

It is beneficial to use character classes whenever possible because they adapt better to non-English locales (including some necessary accented characters, etc.).
Advanced Substitution Features
We have seen how to perform simple and even somewhat complex direct substitutions, but sed can do much more. In fact, you can reference part or all of the matched regular expression and use these parts to construct the replacement string. As an example, suppose you are replying to a message. The following example will prepend the phrase "ralph said: " to each line:


$ sed -e 's/.*/ralph said: &/' origmsg.txt


The output is as follows:


ralph said: Hiya Jim, ralph said: ralph said:
I sure like this sed stuff! ralph said:


The '&' character is used in the replacement string of this example, which tells sed to insert the entire matched regular expression. Therefore, you can insert anything matched by '.*' (the largest group or entire line of zero or more characters in the line) anywhere in the replacement string, even multiple times. This is very good, but sed is even more powerful.
Those Wonderful Parentheses with Backslashes
The 's///' command is even better than '&' because it allows us to define regions in the regular expression, which can then be referenced in the replacement string. As an example, suppose you have a file containing the following text:


foo bar oni eeny meeny miny larry curly moe jimmy the weasel


Now suppose you want to write a sed script that will replace "eeny meeny miny" with "Victor eeny-meeny Von miny" and so on. To do this, first write a regular expression that is separated by spaces and matches three strings.


'.* .* .*'


Now, enclose each area of interest with backslash-enclosed parentheses to define the areas:


'\(.*\) \(.*\) \(.*\)'


This regular expression works the same as the first regular expression, except that it defines three logical areas that can be referenced in the replacement string. Here is the final script:


$ sed -e 's/\(.*\) \(.*\) \(.*\)/Victor \1-\2 Von \3/' myfile.txt


As you can see, each area delimited by parentheses is referenced by entering '\x' (where x is the area number starting from 1). Enter as follows:


Victor foo-bar Von oni Victor eeny-meeny Von miny Victor larry-curly Von moe Victor jimmy-the Von weasel


As you become more familiar with sed, you can do quite powerful text processing with minimal effort. You might be wondering how you would handle such a problem with a familiar scripting language -- can you easily implement such a solution in one line of code?
Combining Commands
When you start creating more complex sed scripts, you need to be able to enter multiple commands. There are several ways to do this. First, you can use semicolons between commands. For example, the following command series uses the '=' command and the 'p' command. The '=' command tells sed to print the line number, and the 'p' command explicitly tells sed to print the line (because it is in '-n' mode).


$ sed -n -e '=;p' myfile.txt


Whenever you specify two or more commands, each command is applied to each line of the file in sequence. In the above example, the '=' command is first applied to line 1, then the 'p' command is applied. Then, sed continues to process line 2 and repeats the process. Although semicolons are convenient, they don't work properly in some situations. Another alternative is to use two -e options to specify two different commands:


$ sed -n -e '=' -e 'p' myfile.txt


However, when using more complex append and insert commands, even multiple '-e' options won't help us. For complex multi-line scripts, the best approach is to put the commands in a separate file. Then, reference the script file with the -f option:


$ sed -n -f mycommands.sed myfile.txt


This approach may be less convenient, but it always works.
Multiple Commands for One Address
Sometimes, you may want to specify multiple commands to be applied to one address. This is especially convenient when performing many 's///' to transform words and syntax in a source file. To execute multiple commands on one address, enter the sed commands in a file, and then group these commands using '{}' characters, as follows:


1,20{ s/inux/GNU\/Linux/g s/samba/Samba/g s/posix/POSIX/g }


The above example will apply three substitution commands to lines 1 to 20 (inclusive). You can also use regular expression addresses or a combination of both:


1,/^END/{ s/inux/GNU\/Linux/g s/samba/Samba/g s/posix/POSIX/g p }


This example will apply all commands between '{}' to all lines starting from line 1 and ending with the line starting with the letter "END" (if "END" is not found in the source file, it will end at the end of the file).
Appending, Inserting, and Changing Lines
Now that we are writing sed scripts in a separate file, we can take advantage of the append, insert, and change line commands. These commands will insert a line after the current line, insert a line before the current line, or replace the current line in the pattern space. They can also be used to insert multiple lines into the output. The insert line command is used as follows:


i\ This line will be inserted before each line


If you don't specify an address for this command, it will be applied to each line and produce output like the following:


This line will be inserted before each line line 1 here
This line will be inserted before each line line 2 here
This line will be inserted before each line line 3 here
This line will be inserted before each line line 4 here


If you want to insert multiple lines before the current line, you can add additional lines by appending a backslash after the previous line, as follows:


i\ insert this line\ and this one\ and this one\ and, uh, this one too.


The append command is used similarly, but it will insert one or more lines after the current line in the pattern space. Its usage is as follows:


a\ insert this line after each line. Thanks! :)


On the other hand, the "change line" command will actually replace the current line in the pattern space, and its usage is as follows:


c\ You're history, original line! Muhahaha!


Because the append, insert, and change line commands require multiple lines of input, you will enter them into a text sed script and then tell sed to execute them by using the '-f' option. Using other methods to pass commands to sed will cause problems.

General Thread -- sed Examples, Part 3


Content:

Robust sed

Text Transformation

Reversing Lines

Reversal Explanation

sed QIF Magic

The Story of Two Formats

Starting the Process

Refinement

Ending Attempts

Don't Get Confused

Resources

About the Author

Rating for this Article





October 2001

In this concluding article of the sed series, Daniel Robbins takes you through the true power of sed. After introducing several important sed scripts, he will demonstrate the writing of some basic sed scripts by converting a Quicken .QIF file into a readable text format. This conversion script is not only practical, but also an excellent example of demonstrating sed scripting capabilities.
Robust sed
In the second sed article, I provided some examples to demonstrate how sed works, but few of them actually do anything particularly useful. In this final article of the sed series, I will change that and use sed to do something practical. I will show you a few examples that not only demonstrate sed's capabilities, but also do something really clever (and convenient). For example, later in this article, I will show you how to design a sed script to convert a .QIF file from Intuit's Quicken financial program into a well-formatted text file. Before doing that, we will look at some less complex but useful sed scripts.

Text Transformation
The first practical script will convert UNIX-style text to DOS/Windows format. As you may know, text files based on DOS/Windows have a CR (carriage return) and LF (line feed) at the end of each line, while UNIX text has only a line feed. Sometimes you may need to move some UNIX text to a Windows system, and this script will perform the necessary format conversion for you.


$ sed -e 's/$/\r/' myunix.txt > mydos.txt


In this script, the '$' regular expression will match the end of the line, and '\r' tells sed to insert a carriage return before it. Inserting a carriage return before the line feed immediately causes each line to end with CR/LF. Please note that '\r' will be replaced with CR only if you are using GNU sed 3.02.80 or later. If you haven't installed GNU sed 3.02.80 yet, see the instructions on how to do so in my first sed article.

I can't remember how many times I have downloaded some example scripts or C code only to find that it is in DOS/Windows format. Although many programs don't care about CR/LF text files in DOS/Windows format, a few programs do -- the most famous one is bash, which will have problems as soon as it encounters a carriage return. The following sed call will convert text in DOS/Windows format to reliable UNIX format:


$ sed -e 's/.$//' mydos.txt > myunix.txt


The script works simply: the substitution regular expression matches the last character of a line, which happens to be the carriage return. We replace it with an empty character, thus removing it completely from the output. If you use this script and notice that you have removed the last character of each line in the output, then you have specified a text file that is already in UNIX format. There is no need to do that!
Reversing Lines
Here is another convenient little script. Like the "tac" command included with most Linux distributions, this script will reverse the order of lines in a file. The name "tac" may be misleading because "tac" does not reverse the position of characters in a line (left and right), but rather reverses the position of lines in a file (top and bottom). Using "tac" to process the following file:


foo bar oni


....will produce the following output:


oni bar foo


You can achieve the same purpose with the following sed script:


$ sed -e '1!G;h;$!d' forward.txt > backward.txt


If you log in to a FreeBSD system that happens to not have the "tac" command, you will find that this sed script is very useful. Although convenient, it is better to know why this script works that way. Let's discuss it.
Reversal Explanation
First, this script contains three separate sed commands separated by semicolons: '1!G', 'h', and '$!d'. Now, we need to understand the addresses used for the first and third commands. If the first command were '1G', then the 'G' command would only be applied to the first line. However, there is an '!' character -- this '!' character ignores the address, that is, the 'G' command will be applied to all lines except the first line. The '$!d' command is similar. If the command were '$d', then the 'd' command would only be applied to the last line in the file ('$' address is an easy way to specify the last line). However, with the '!', '$!d' will apply the 'd' command to all lines except the last line. Now, what we need to understand is what these commands themselves do.

When the reversal script is executed on the above text file, the first command to be executed is 'h'. This command tells sed to copy the contents of the pattern space (the buffer that holds the current line being processed) to the hold space (a temporary buffer). Then, the 'd' command is executed, which deletes "foo" from the pattern space so that it is not printed after all commands are executed for this line.

Now, for the second line. After "bar" is read into the pattern space, the 'G' command is executed, which appends the contents of the hold space ("foo\n") to the pattern space ("bar\n"), making the contents of the pattern space "bar\n\foo\n". The 'h' command puts this content back into the hold space for protection, and then 'd' deletes the line from the pattern space so that it is not printed.

For the last "oni" line, the same steps are repeated except that the contents of the pattern space (three lines) are not deleted by 'd' (due to '$!') and are printed to standard output.

Now, you can use sed to perform some powerful data transformations.
sed QIF Magic
For the past few weeks, I have been wanting to buy a copy of Quicken to balance my bank account. Quicken is a very good financial program and will certainly do the job successfully. However, after thinking about it, I felt that I could easily write some software to balance my checkbook. I thought, after all, I am a software developer!

I developed a nice small checkbook balancing program (using awk) that calculates the balance by analyzing the syntax of a text file containing all my transactions. After a slight adjustment, I improved it so that I could track different loan and borrowing categories like Quicken. But I also wanted to add a feature. Recently, I transferred my account to a bank that has an online Web account interface. One day, I noticed that this bank's Web site allows me to download my account information in Quicken's .QIF format. I immediately thought it would be great if I could convert this information into text format.
The Story of Two Formats
Before looking at the QIF format, let's look at my checkbook.txt format:


28 Aug 2000 food - - Y Supermarket 30.94 25 Aug 2000 watr - 103 Y Check 103 52.86


In my file, all fields are separated by one or more tabs, and each transaction occupies one line. The next field after the date lists the expenditure type (if it is an income item, it is "-"). The third field lists the income type (if it is an expenditure item, it is "-"). Then, there is a check number field (if it is empty, it is still "-"), a transaction completion field ("Y" or "N"), a comment, and a dollar amount field. Now, let's look at the QIF format. When you view the downloaded QIF file with a text viewer, it looks like the following:


!Type:Bank D08/28/2000 T-8.15 N PCHECKCARD SUPERMARKET ^ D08/28/2000 T-8.25 N PCHECKCARD PUNJAB RESTAURANT ^ D08/28/2000 T-17.17 N PCHECKCARD SUPERMARKET


After browsing the file, it's not difficult to guess its format -- ignoring the first line, the rest are formatted as follows:



D<data>
T<transaction amount>
N<check number>
P<description>
^ (This is the field separator)



Starting the Process
When dealing with such an important sed project, don't get discouraged -- sed allows you to gradually modify the data into the final form. As you go along, you can continue to refine the sed script until the output is exactly as you expect. There is no need to guarantee it to be completely correct the first time you try.

To get started, first create a file named "qiftrans.sed" and start modifying the data:


1d /^^/d s/]//g


The first '1d' command deletes the first line, the second command removes those annoying '^' characters from the output. The last line removes any control characters that may be present in the file. Since we are dealing with an external file format, I want to eliminate the risk of encountering any control characters along the way. So far, so good. Now, add some processing functions to this basic script:


1d /^^/d s/]//g /^D/ {
s/^D\(.*\)/\1\tOUTY\tINNY\t/
s/^01/Jan/ s/^02/Feb/
s/^03/Mar/ s/^04/Apr/
s/^05/May/ s/^06/Jun/
s/^07/Jul/ s/^08/Aug/
s/^09/Sep/ s/^10/Oct/
s/^11/Nov/ s/^12/Dec/
s:^\(.*\)/\(.*\)/\(.*\):\2 \1 \3: }


First, add a '/^D/' address so that sed will only start processing when it encounters the first character 'D' of the QIF data field. When sed reads such a line into its pattern space, it will execute all commands in the braces in sequence.

The first command in the braces will transform a line like this:


D08/28/2000


into:


08/28/2000 OUTY INNY


Of course, the format is not perfect yet, but that's okay. We will gradually refine the contents of the pattern space as we go. The final effect of the next 12 lines is to transform the data into a three-letter format, and the last line removes the three slashes from the data. Finally, we get this line:


Aug 28 2000 OUTY INNY


The OUTY and INNY fields are placeholders that will be replaced later. We can't determine them yet because if the dollar amount is negative, we will set OUTY and INNY to "misc" and "-", but if the dollar amount is positive, we will change them to "-" and "inco" respectively. Since we haven't read the dollar amount yet, we need to use placeholders for now.
Refinement
Now further refine:


1d /^^/d s/]//g /^D/ {
s/^D\(.*\)/\1\tOUTY\tINNY\t/
s/^01/Jan/ s/^02/Feb/
s/^03/Mar/ s/^04/Apr/
s/^05/May/ s/^06/Jun/
s/^07/Jul/ s/^08/Aug/
s/^09/Sep/ s/^10/Oct/
s/^11/Nov/ s/^12/Dec/
s:^\(.*\)/\(.*\)/\(.*\):\2 \1 \3:
N N N
s/\nT\(.*\)\nN\(.*\)\nP\(.*\)/NUM\2NUM\t\tY\t\t\3\tAMT\1AMT/
s/NUMNUM/-/ s/NUM\(*\)NUM/\1/
s/\(\),/\1/ }


The last seven lines are a bit complex, so let's discuss them in detail. First, use three consecutive 'N' commands. The 'N' command tells sed to read the next line into the input and then append it to the current pattern space. These three 'N' commands cause the next three lines to be appended to the current pattern space buffer, and now this line looks like this:


28 Aug 2000 OUTY INNY \nT-8.15\nN\nPCHECKCARD SUPERMARKET


sed's pattern space becomes messy -- we need to remove the extra newlines and perform some additional formatting. To do this, we will use the substitution command. The pattern to match is:


'\nT.*\nN.*\nP.*'


This will match a newline followed by 'T', zero or more characters, a newline, 'N', any number of characters, a newline, 'P', and any number of characters. Yikes! This regular expression will match the entire content of the three lines just appended to the pattern space. But we want to reformat this area, not replace it entirely. The dollar amount, check number (if any), and description need to appear in the replacement string. To do this, we enclose those "interesting parts" with backslash-enclosed parentheses so that we can reference them in the replacement string (using '\1', '\2', and '\3' to tell sed where to insert them). Here is the final command:


s/\nT\(.*\)\nN\(.*\)\nP\(.*\)/NUM\2NUM\t\tY\t\t\3\tAMT\1AMT/


This command transforms our line into:


28 Aug 2000 OUTY INNY NUMNUM Y CHECKCARD SUPERMARKET AMT-8.15AMT


Although the line is getting better, there are a few things that look a bit... ah... interesting. First is that stupid "NUMNUM" string -- what's its purpose? If you look at the last two lines of the sed script, you will find its purpose. The last two lines replace "NUMNUM" with "-" and replace "NUM"<number>"NUM" with <number>. As you can see, enclosing the check number with stupid markers allows us to easily insert a "-" when this field is empty.
Ending Attempts
The last line removes the comma after the number. It transforms a dollar amount like "3,231.00" into the format "3231.00" that I use. Now, let's look at the final script:

Final "QIF to Text" Script
1d /^^/d s/]//g /^D/ { s/^D\(.*\)/\1\tOUTY\tINNY\t/
s/^01/Jan/ s/^02/Feb/ s/^03/Mar/ s/^04/Apr/ s/^05/May/
s/^06/Jun/ s/^07/Jul/ s/^08/Aug/ s/^09/Sep/ s/^10/Oct/
s/^11/Nov/ s/^12/Dec/ s:^\(.*\)/\(.*\)/\(.*\):\2 \1 \3:
N N N s/\nT\(.*\)\nN\(.*\)\nP\(.*\)/NUM\2NUM\t\tY\t\t\3\tAMT\1AMT/
s/NUMNUM/-/ s/NUM\(*\)NUM/\1/ s/\(\),/\1/
/AMT-*.*AMT/b fixnegs
s/AMT\(.*\)AMT/\1/ s/OUTY/-/ s/INNY/inco/
b done :fixnegs s/AMT-\(.*\)AMT/\1/ s/OUTY/misc/
s/INNY/-/ :done }



The additional eleven lines use substitution and some branching functions to beautify the output. First, look at this line:


/AMT-*.*AMT/b fixnegs


This line contains a branch command in the format "/regexp/b label". If the pattern space matches the regular expression, sed will branch to the fixnegs label. You should be able to easily find this label, which is ":fixnegs" in the code. If the regular expression does not match, processing continues in the normal way.

Now that you understand how the command itself works, let's look at the branch. If you look at the branch regular expression, you will see that it matches a string that is followed by '-' followed by any number of digits, a '.', any number of digits, and 'AMT' after 'AMT'. As I'm sure you've guessed, this regular expression is specifically for handling negative dollar amounts. Before this, the dollar amount is enclosed in 'ATM' so that it can be easily found later. Since the regular expression only matches dollar amounts starting with '-', this branch only occurs when borrowing happens. If you are tracking a loan, you should set OUTY to 'misc' and INNY to '-', and you should remove the negative sign in front of the loan amount. If you follow the flow of the code, you will see that this is exactly what happens. If the branch is not executed, replace OUTY with '-' and INNY with 'inco'. Done! Now the output line is perfect:


28 Aug 2000 misc - - Y CHECKCARD SUPERMARKET -8.15


Don't Get Confused
As you can see, converting data with sed is not that difficult as long as you solve the problem step by step. Don't try to use one sed command or solve all problems at once. Instead, proceed towards the goal step by step and continuously refine the sed script until its output is exactly as you want. sed has many features, and I hope you are already very familiar with its internal working principles and continue to work hard to master it further!

Resources
You can refer to the English original of this article on the developerWorks global site at

English Original

.
Read other sed articles by Daniel on developerWorks: General Thread: sed Examples,

Part 2

and

Part 3

.
View Eric Pement's excellent

sed FAQ

.
You can find sed 3.02 resources at

ftp://ftp.gnu.org/pub/gnu/sed

.
You will find the excellent new sed 3.02.80 at

alpha.gnu.org

.
In addition, Eric Pement also has some convenient

sed one-liners

, which any aspiring sed master should take a look at.
If you want to read a good old-fashioned book, O'Reilly's

sed & awk, 2nd Edition

will be an excellent choice.
You may want to read the

7th edition UNIX's sed man page

(approximately 1978!).
Read Felix von Leitner's short

sed tutorial

.
Read David Mertz's

"Text processing in Python"

on developerWorks.
About Regular Expressions:
Review in

Using Regular Expressions

, find and modify the patterns in this free dW exclusive tutorial text.
View the regular expression

how-to document

at Python.org.
Refer to the

overview of regular expressions

at the University of Kentucky in the United States.

Last edited by 无奈何 on 2006-10-26 at 01:54 PM ]

Repost Note: Original link http://bbs.chinaunix.net/archiver/?tid-691881.html

sed Usage:
sed 'Command' filename(s) Only display the result without modifying the file.

1、sed '2,5d' file Display file, removing lines 2-5, but no error is reported when the number of lines exceeds the actual number of lines in the file.
sed '/10/d' file Display file, removing lines containing 101-104.
sed '2,$d' file Display file, only display the first line. sed '2,$!d' file only displays other lines except the first line.
sed '/^ *$/d file Delete blank lines in the file.
2、sed -n '/10/p' file Only display lines containing 101-104 in file. (-n and p must be used together, otherwise only p will display the entire file and display the found line one more time)
sed -n '5p' file Only display the 5th line of the file
3、sed 's/moding/moden/g' file Replace moding with moden
4、sed -n 's/^west/north/p' file Replace lines starting with west with north and display them.
5、sed 's/$/&.5/' file Replace lines ending with 3 digits in file with the original digits plus ".5", & represents the searched string.
6、sed 's/\(mod\)ing/\1en/g file Enclose mod as pattern 1 in parentheses, then replace.
sed 's/...$//' file Delete the last three characters of each line.
sed 's/^...//' file Delete the first three characters of each line.
7、sed 's#moding#moden#g' file Replace moding with moden, # after s represents the delimiter between the search string and the replacement string.
8、sed -n '/101/,/105/p' file Display from the matching line of 101 to the matching line of 105. If only the matching line of 101 is found, then from the matching line of 101 to the end of the file.
sed -n '2,/999/p' file Display from line 2 to the matching line.
9、sed '/101/,/105/s/$/ 20050119/' file Add " 20050119" content at the end of lines from the matching line of 101 to the matching line of 105.
10、sed -e '1,3d' -e 's/moding/moden/g' file First delete lines 1-3 of the file, then perform replacement.
sed -e '/^#/!d' file Display lines starting with # in the file.
11、sed '/101/r newfile' file Add the content of file newfile at each matching line
sed '/101/w newfile' file Write matching lines to newfile.
12、sed '/101/a\
> ###' file Add a new line after the matching line.
sed '/101/i\
> ###' file Add a new line before the matching line.
sed '/101/c\
> ###' file Replace the matching line with a new line.

13、sed 'y/abcd/ABCD/' file Replace a, b, c, d with ABCD respectively.
14、sed '5q' file Exit when displaying to line 5.
15、sed '/101/{ n; s/moding/moden/g; }' file After finding the matching line in the file, perform replacement on the next line (n).
sed '/101/{ s/moding/moden/g; q; }' file After finding the first matching line in the file, perform replacement and then exit.
16、sed -e '/101/{ h; d; }' -e '/104/{ G; }' file After finding the matching line with 101 in the file, first store it in a cache, then place it after the matching line with 104.
sed -e '/101/{ h; d; }' -e '/104/{ g; }' file After finding the matching line with 101 in the file, first store it in a cache, then replace the matching line with 104.
sed -e '/101/h' -e '$G' file Place the last matching line at the end of the file.
sed -e '/101/h' -e '$g' file Replace the last line of the file with the last matching line.
sed -e '/101/h' -e '/104/x' file After finding the matching line with 101 in the file, first store it in a cache, then swap it with the matching line with 104.
17、sed -f sfile file Operate according to the command list in file sfile.
cat sfile
/101/a\
####101####\
****101****
/104/c\
####104 deleted####\
****104 deleted****
1i\
####test####\
****test****

Last edited by 无奈何 on 2006-10-26 at 02:03 PM ]

Repost Note: Original link http://blah.blogsome.com/2006/04/01/sed_iter/

Using Loops in sed

Loops can usually be written in the form of conditional jumps. sed does not have a dedicated loop statement, but it provides jump commands, so we can still implement loops. This article summarizes several ways to use sed for looping. The way sed processes text itself is a kind of loop:

do while not EOF
read line
... do sth
end do.


1 Making Judgments in sed

Because sed only processes characters and line numbers, it can only make string matching judgments or line number judgments through patterns. So the judgment conditions need to appear in the form of strings or line numbers.

1.1 Storing Flag Bits in holdspace

Store a string as a flag bit in hs, such as:


# Perform 6 operations
1{x; s/^/654321/; x}
:a
x;
/./{ s/.//; x; s/reg/ex/; ba}
x;
do sth else;

# Perform 6 replacement operations on each line
1{x; s/^/654321/; x}
G
:a
/\n./{ s/reg/ex/; s/\n.$//; tb; s/.$//; ba}
:b
do sth else

1.2 Storing Flag Bits in pattern space

Use pattern space (the flag bit is attached to the front or back), which is basically the same as the hs method except that the flag bit is placed in ps.

1{ s/^/654321\n/ }
:a
/.\n/{ s/.//; s/reg/ex/; ba }
do sth else

1.3 Judging with Addresses

Addresses (commonly used addresses are 1,$). When the loop condition is related to addresses or line numbers, this method can be used.

sed '/./{H;d};x;/re/p' # Display a certain paragraph
2,8{H;d}; $G # Similar to :2,8m$ in ed. d has two roles here: a clear PS. b force to enter the next cycle.

1.4 Judging with Patterns

Use the content of the current ps as the standard for judgment. When the loop condition is related to the input content, this method can be used. This method is similar to the flag bit method. The difference is that we do not artificially set the flag but use the current ps content as the flag. Please refer to the following example:

:a
do sth with regexp
/regexp/ba


If there are s commands in the middle, we often use t to jump, so the above can be written as:
:a
s/regexp/blah/
ta

2 Command Analysis Commonly Used in Loops

q, b, t, T, d, n, N, :label

2.1 b/t Control Loops

b is a branch (jump) command in sed. Its format is `b label`, and it can also be written as `blabel` without the middle space. The function of the above command is to continue executing the script from :label. The label must be defined in the script by adding a colon (:) in front, such as `:loop` `:lable`. Most seds have length restrictions on labels. For specific restrictions, refer to the sed faq. If b is not followed by a label, it defaults to jumping to the end of the script.
It should be noted that many seds do not allow other commands to be followed, so:
gsed 'b abc;s/^/eee/;:abc;…'
Such a command in other versions of sed should be written as:

sed 'b abc
s/^/eee/
:abc'
or:
sed -e 'b abc' -e 's/^/eee/; :abc'


The t command is similar to b. The difference is that t determines whether to jump based on whether the previous s command is successful. If it is successful, it jumps. If multiple conditional jumps are used after one s command, the second and subsequent ts will all fail. GNU sed also provides the T command opposite to t.

$ echo abc|sed 's/a/a/;bb;:b;tc;s/^/zzz/;:c'
abc

$ echo abc|sed 's/a/a/;tb;:b;tc;s/^/zzz/;:c'
zzzabc

$ echo abc|sed 's/a/a/;tb;:b;tc;s/^/zzz/;:c'
zzzabc

b/t control loop:
s/re/&/;t
s/re/&/;T
is equivalent to
/re/b
/re/!b
But in the following content, we will see that the combination of s/t can be used in some more complex situations.

When using b for looping, the following method is often used to exit the loop:
Use a pattern or address (loop condition) before b:

:a
...
/regexp/ba


In the above case, it is usually required that the intermediate statements modify /regexp/ or have another exit command, otherwise it will become an infinite loop.

2.1.0.1 Exit Conditions

When using b/t loops, in order to avoid infinite loops, exit conditions are usually set. Such as:

:a
...
/xxx/b
...
/regexp/ba

In addition to using b, t, N, q, d can also be used as exit commands, and patterns or line numbers can be used as exit conditions. When there is a successful replacement before, `t` can be used to jump to any position in the script - of course, it can also jump out of the loop body. Using N in the last line will exit the script. Using this, we can exit the loop body. However, the results of using `N` in the last line are different in different versions of sed - some will display the content of ps, and some will not. The `q` command is used to exit the script, so it will not loop again. The side effects of the `d` and `D` commands are to force the script to enter the next cycle, which allows us to use these two commands to form a loop between lines. If the next cycle does not meet the entry conditions of the loop, the loop stops - so these two commands are both commands for looping and exiting commands. Of course, patterns or line numbers can be used as conditions for running commands in front of these lines.

If the intermediate statement does not modify /regexp/, the result is similar:

:a
...
ba


Example: To delete all xxx after the # number in the input.

Input:
abc#efxxxghxxxxijxxx
Output:
abc#efghxij

sed ':a
s/\(#.*\)xxx/\1/ # Modify the loop flag
/#.*xxx/ba'


Of course, then t can come in handy:

sed ':a; s/\(#.*\)xxx/\1/; ta'

Here are some examples:

echo -e "abc\nefg" | sed ':a;/re/b;ba'
echo -e "abc\nefg" | sed ':a;/re/!b;ba'
echo -e "abc\nefg" | sed ':a;s/re/&/;t;ba'
echo -e "abc\nefg" | gsed ':a;s/re/&/;T;ba'
To control the loop:
echo -e "abc\nefg" | sed ':a;/re/ba'
echo -e "abc\nefg" | sed ':a;/re/!ba'
echo -e "abc\nefg" | sed ':a;s/re/&/;ta'
echo -e "abc\nefg" | sed ':a;s/re/&/;Ta'
echo -e "abc\nefg" | sed '/re/b; :'

2.2 d/D Control Loops

Using d to control the loop is also a relatively common technique in sed scripts. The reason why d can be used to control the loop is mainly because after deleting, it will not execute the following commands and will directly enter the next cycle. Example:

sed 'd; s/^/abc/' # The s command will not be executed

2.3 Flag Bit Control Loop

We have already talked about some methods for making conditional judgments in sed. These methods can be used not only as conditions for entering the loop but also as conditions for exiting the loop. Here is an example:

G;s/$/123456789/ # Loop 9 times
:loop
s/\n$//;t break # Exit the loop
... do sth # Perform operations
s/.$// # Decrement by one
b loop # next

2.4 N as Exit Condition

N adds the next line to the current PS. If N is run on the last line, sed usually exits. If used in a loop, it will exit the loop. It should be noted that the processing of the last line by different Ns is different. Some versions will exit quietly when N is executed on the last line. While other versions such as GNU sed will default to displaying the content of PS and then exiting.
See the following example:

echo -e "abc\nefg" | sed ':a;ba'
echo -e "abc\nefg" | sed ':a;n;ba'
echo -e "abc\nefg" | sed ':a;N;ba'
echo -e "abc\nefg" | sed ':a;$!n;ba'
echo -e "abc\nefg" | sed ':a;$!N;ba'


In the first example, sed will enter an infinite loop. The second and third examples will exit normally. The second example will display the input, and the behavior of the third example is related to the version of sed. GNU sed will display the input. The last example will enter an infinite loop. As mentioned earlier, $!N can make N display the result in all versions of sed. But still need to judge the timing of use.

3 Examples

There are many examples of using loops in sed. Here are two examples.

# An example in "sed one-line script"
# Right-align all text with a width of 79 characters
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # 78 characters plus a final space

# Move lines 2 to 8 to the end of the file
# Similar to :2,8m$ in ed.
# d has two roles here: a. clear PS. b. force to enter the next cycle.
# do while 2<=linenum<=8; H; end do
sed '2,8{H;d}; $G'




Last edited by 无奈何 on 2006-10-26 at 02:08 PM ]

sed is really powerful~ : )

GNU stuff is really good.

It's a pity that you don't learn such a powerful command, which is a disservice to the author.................. Alas

　　Looking at such a long passage, my head is already big... Hey

SED is very powerful, but it's like looking at天书. I'll study it later when I have time.

Here I found another article that should be similar.

Original text and other language viewing address:
http://sed.sourceforge.net/sed1line_zh-CN.html

SED One-Line Script Quick Reference

-------------------------------------------------------------------------
SED One-Line Script Quick Reference (Unix Stream Editor) December 29, 2005

English title: USEFUL ONE-LINE SCRIPTS FOR SED (Unix stream editor)
Original title: HANDY ONE-LINERS FOR SED (Unix stream editor)

Organizer: Eric Pement - Email: pementenorthparkedu Version 5.5
Translator: Joe Hong - Email: hq00e126com

The latest (English) version of this document can be found at the following address:
http://sed.sourceforge.net/sed1line.txt
http://www.pement.org/sed/sed1line.txt

Other language versions:
Chinese - http://sed.sourceforge.net/sed1line_zh-CN.html
Czech - http://sed.sourceforge.net/sed1line_cz.html
Dutch - http://sed.sourceforge.net/sed1line_nl.html
French - http://sed.sourceforge.net/sed1line_fr.html
German - http://sed.sourceforge.net/sed1line_de.html

Portuguese - http://sed.sourceforge.net/sed1line_pt-BR.html


Text spacing:
--------

# Add a blank line after each line
sed G

# Delete all original blank lines and add a blank line after each line.
# This way, there will be exactly one blank line after each line in the output text.
sed '/^$/d;G'

# Add two blank lines after each line
sed 'G;G'

# Delete all blank lines generated by the first script (i.e., delete all even lines)
sed 'n;d'

# Insert a blank line before the line matching the pattern "regex"
sed '/regex/{x;p;x;}'

# Insert a blank line after the line matching the pattern "regex"
sed '/regex/G'

# Insert a blank line before and after the line matching the pattern "regex"
sed '/regex/{x;p;x;G;}'

Numbering:
--------

# Number each line in the file (simple left-aligned style). Here, a "tab" is used
# (tab, see the description of the usage of '\t' at the end of this article) instead of a space to align the edges.
sed = filename | sed 'N;s/\n/\t/'

# Number all lines in the file (line number on the left, text right-aligned).
sed = filename | sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /'

# Number all lines in the file, but only display the line numbers of non-blank lines.
sed '/./=' filename | sed '/./N; s/\n/ /'

# Count the number of lines (simulate "wc -l")
sed -n '$='

Text conversion and substitution:
--------

# Unix environment: Convert DOS newline characters (CR/LF) to Unix format.
sed 's/.$//' # Assuming all lines end with CR/LF
sed 's/^M$//' # In bash/tcsh, press Ctrl-V instead of Ctrl-M
sed 's/\x0D$//' # ssed, gsed 3.02.80, and later versions

# Unix environment: Convert Unix newline characters (LF) to DOS format.
sed "s/$/`echo -e \\\r`/" # Command used under ksh
sed 's/$'"/`echo \\\r`/" # Command used under bash
sed "s/$/`echo \\\r`/" # Command used under zsh
sed 's/$/\r/' # gsed 3.02.80 and later versions

# DOS environment: Convert Unix newline characters (LF) to DOS format.
sed "s/$//" # Method 1
sed -n p # Method 2

# DOS environment: Convert DOS newline characters (CR/LF) to Unix format.
# The following script is only valid for UnxUtils sed 4.0.7 and later versions. To identify the UnxUtils version of
# sed, you can use its unique "--text" option. You can use the help option ("--help") to see
# if there is a "--text" item in it to determine whether the used version is UnxUtils. Other DOS
# versions of sed cannot perform this conversion. But it can be achieved with "tr".
sed "s/\r//" infile >outfile # UnxUtils sed v4.0.7 or later
tr -d \r <infile >outfile # GNU tr 1.22 or later

# Delete leading "whitespace characters" (spaces, tabs) from each line
# Make it left-aligned
sed 's/^*//' # See the description of the usage of '\t' at the end of this article

# Delete trailing "whitespace characters" (spaces, tabs) from each line
sed 's/*$//' # See the description of the usage of '\t' at the end of this article

# Delete leading and trailing whitespace characters from each line
sed 's/^*//;s/*$//'

# Insert 5 spaces at the beginning of each line (move the entire text 5 characters to the right)
sed 's/^/ /'

# Right-align all text with a width of 79 characters
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # 78 characters plus a final space

# Center all text with a width of 79 characters. In method 1, spaces are filled at the beginning and end of each line for centering. In method 2, spaces are only filled in front of the text during centering, and finally half of these spaces will be deleted. In addition, no spaces are filled at the end of each line.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # Method 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' # Method 2

# Find the string "foo" in each line and replace the found "foo" with "bar"
sed 's/foo/bar/' # Only replace the first "foo" string in each line
sed 's/foo/bar/4' # Only replace the fourth "foo" string in each line
sed 's/foo/bar/g' # Replace all "foo" in each line with "bar"
sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # Replace the second-to-last "foo"
sed 's/\(.*\)foo/\1bar/' # Replace the last "foo"

# Only replace "foo" with "bar" if the string "baz" appears in the line
sed '/baz/s/foo/bar/g'

# Replace "foo" with "bar" and only replace it if the string "baz" does not appear in the line
sed '/baz/!s/foo/bar/g'

# Replace "scarlet", "ruby", or "puce" with "red" regardless
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # Valid for most seds
gsed 's/scarlet\|ruby\|puce/red/g' # Only valid for GNU sed

# Reverse all lines, with the first line becoming the last line, and so on (simulate "tac").
# For some reason, HHsed v1.5 will delete blank lines in the file when using the following command
sed '1!G;h;$!d' # Method 1
sed -n '1!G;h;$p' # Method 2

# Reverse the characters in the line, with the first word becoming the last word, etc. (simulate "rev")
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'

# Concatenate every two lines into one line (similar to "paste")
sed '$!N;s/\n/ /'

# If the current line ends with a backslash "\", append the next line to the end of the current line
# And remove the original line ending backslash
sed -e :a -e '/\\$/N; s/\\\n//; ta'

# If the current line starts with an equal sign, append the current line to the end of the previous line
# And replace the original line start "=" with a single space
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'

# Add comma separators to numeric strings, changing "1234567" to "1,234,567"
gsed ':a;s/\B\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/\(.*\)\(\{3\}\)/\1,\2/;ta' # Other seds

# Add comma separators to values with decimal points and minus signs (GNU sed)
gsed -r ':a;s/(^|)(+)({3})/\1\2,\3/g;ta'

# Add a blank line after every 5 lines (add a blank line after lines 5, 10, 15, 20, etc.)
gsed '0~5G' # Only valid for GNU sed
sed 'n;n;n;n;G;' # Other seds

Selectively display specific lines:
--------

# Display the first 10 lines of the file (simulate the behavior of "head")
sed 10q

# Display the first line of the file (simulate "head -1" command)
sed q

# Display the last 10 lines of the file (simulate "tail")
sed -e :a -e '$q;N;11,$D;ba'

# Display the last 2 lines of the file (simulate "tail -2" command)
sed '$!N;$!D'

# Display the last line of the file (simulate "tail -1")
sed '$!d' # Method 1
sed -n '$p' # Method 2

# Display the second-to-last line of the file
sed -e '$!{h;d;}' -e x # Outputs a blank line when there is only one line in the file
sed -e '1{$q;}' -e '$!{h;d;}' -e x # Displays the line when there is only one line in the file
sed -e '1{$d;}' -e '$!{h;d;}' -e x # Does not output when there is only one line in the file

# Only display lines matching the regular expression (simulate "grep")
sed -n '/regexp/p' # Method 1
sed '/regexp/!d' # Method 2

# Only display lines that "do not" match the regular expression (simulate "grep -v")
sed -n '/regexp/!p' # Method 1, corresponding to the previous command
sed '/regexp/d' # Method 2, similar syntax

# Find "regexp" and display the line before the matching line, but do not display the matching line
sed -n '/regexp/{g;1!p;};h'

# Find "regexp" and display the line after the matching line, but do not display the matching line
sed -n '/regexp/{n;p;}'

# Display the line containing "regexp" and its preceding and following lines, and add the line number of the line where "regexp" is located before the first line (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h

# Display lines containing "AAA", "BBB", or "CCC" (in any order)
sed '/AAA/!d; /BBB/!d; /CCC/!d' # The order of strings does not affect the result

# Display lines containing "AAA", "BBB", and "CCC" (fixed order)
sed '/AAA.*BBB.*CCC/!d'

# Display lines containing "AAA", "BBB", or "CCC" (simulate "egrep")
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # Most seds
gsed '/AAA\|BBB\|CCC/!d' # Valid for GNU sed

# Display paragraphs containing "AAA" (paragraphs are separated by blank lines)
# HHsed v1.5 must add "G;" after "x;", and the next three scripts are like this
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'

# Display paragraphs containing the three strings "AAA", "BBB", and "CCC" (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'

# Display paragraphs containing any one of the three strings "AAA", "BBB", "CCC" (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\|BBB\|CCC/b;d' # Only valid for GNU sed

# Display lines containing 65 or more characters
sed -n '/^.\{65\}/p'

# Display lines containing fewer than 65 characters
sed -n '/^.\{65\}/!p' # Method 1, corresponding to the above script
sed '/^.\{65\}/d' # Method 2, a simpler method

# Display part of the text - from the line containing the regular expression to the end of the last line
sed -n '/regexp/,$p'

# Display part of the text - specify the line number range (from line 8 to line 12, including lines 8 and 12)
sed -n '8,12p' # Method 1
sed '8,12!d' # Method 2

# Display line 52
sed -n '52p' # Method 1
sed '52!d' # Method 2
sed '52q;d' # Method 3, more efficient when processing large files

# Display every 7th line starting from line 3
gsed -n '3~7p' # Only valid for GNU sed
sed -n '3,${p;n;n;n;n;n;n;}' # Other seds

# Display the text between two regular expressions (including)
sed -n '/Iowa/,/Montana/p' # Case-sensitive way

Selectively delete specific lines:
--------

# Display the entire document except the content between the two regular expressions
sed '/Iowa/,/Montana/d'

# Delete adjacent duplicate lines in the file (simulate "uniq")
# Only keep the first line of duplicate lines, delete other lines
sed '$!N; /^\(.*\)\n\1$/!P; D'

# Delete duplicate lines in the file, regardless of whether they are adjacent. Note the cache size supported by the hold space, or use GNU sed.
sed -n 'G; s/\n/&&/; /^\(*\n\).*\n\1/d; s/\n//; h; P'

# Delete all lines except duplicate lines (simulate "uniq -d")
sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'

# Delete the first 10 lines of the file
sed '1,10d'

# Delete the last line of the file
sed '$d'

# Delete the last two lines of the file
sed 'N;$!P;$!D;$d'

# Delete the last 10 lines of the file
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # Method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # Method 2

# Delete lines that are multiples of 8
gsed '0~8d' # Only valid for GNU sed
sed 'n;n;n;n;n;n;n;d;' # Other seds

# Delete lines matching the pattern
sed '/pattern/d' # Delete lines containing pattern. Of course, pattern
# can be replaced with any valid regular expression

# Delete all blank lines in the file (the same effect as "grep '.' ")
sed '/^$/d' # Method 1
sed '/./!d' # Method 2

# Only keep the first line of multiple adjacent blank lines. And delete blank lines at the top and bottom of the file.
# (Simulate "cat -s")
sed '/./,/^$/!d' # Method 1, delete blank lines at the top of the file, allowing one blank line at the bottom to remain
sed '/^$/N;/\n$/D' # Method 2, allowing one blank line at the top to remain, and no blank line at the bottom to remain

# Only keep the first two lines of multiple adjacent blank lines.
sed '/^$/N;/\n$/N;//D'

# Delete all blank lines at the top of the file
sed '/./,$!d'

# Delete all blank lines at the bottom of the file
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # Valid for all seds
sed -e :a -e '/^\n*$/N;/\n$/ba' # The same as above, but only valid for gsed 3.02.*

# Delete the last line of each paragraph
sed -n '/^$/{p;h;};/./{x;/./p;}'

Special applications:
--------

# Remove nroff marks in man pages. When using the 'echo' command under Unix System V or bash shell, the -e option may be required.
sed "s/.`echo \\\b`//g" # The outer double brackets are necessary (Unix environment)
sed 's/.^H//g' # In bash or tcsh, press Ctrl-V then Ctrl-H
sed 's/.\x08//g' # Hexadecimal representation used by sed 1.5, GNU sed, ssed

# Extract the newsgroup or e-mail header
sed '/^$/q' # Delete all content after the first blank line

# Extract the body part of the newsgroup or e-mail
sed '1,/^$/d' # Delete all content before the first blank line

# Extract the "Subject" (title field) from the mail header and remove the initial "Subject: " words
sed '/^Subject: */!d; s///;q'

# Get the reply address from the mail header
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'

# Get the e-mail address. Further remove non-e-mail address parts from the line of mail header generated by the previous script. (See the previous script)
sed 's/ *(.*)//; s/>.*//; s/.* *//'

# Add an angle bracket and a space at the beginning of each line (quoted information)
sed 's/^/> /'

# Remove the angle bracket and space at the beginning of each line (unquote)
sed 's/^> //'

# Remove most HTML tags (including cross-line tags)
sed -e :a -e 's/<*>//g;/</N;//ba'

# Decode uuencode files divided into multiple volumes. Remove the header information and only keep the uuencode encoding part.
# The file must be passed to sed in a specific order. The first version of the script below can be directly entered on the command line;
# The second version can be put into a shell script with execute permission. (Modified from a script by Rahul Dhesi.)
sed '/^end/,/^begin/d' file1 file2 ... fileX | uudecode # vers. 1
sed '/^end/,/^begin/d' "$@" | uudecode # vers. 2

# Sort the paragraphs in the file alphabetically. Paragraphs are separated by one or more blank lines. GNU sed uses the character "\v" to represent vertical tab, which is used here as a placeholder for the newline character - of course, you can also use other characters not used in the file instead.
sed '/./{H;d;};x;s/\n/={NL}=/g' file | sort | sed '1s/={NL}=//;s/={NL}=/\n/g'
gsed '/./{H;d};x;y/\n/\v/' file | sort | sed '1s/\v//;y/\v/\n/'

# Compress each .TXT file separately, delete the original file after compression, and name the compressed .ZIP file the same as the original (only the extension is different). (DOS environment: "dir /b" shows filenames without paths)
echo @echo off >zipup.bat
dir /b *.txt | sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/" >>zipup.bat


Using SED: Sed accepts one or more editing commands, and each line is read in and these commands are applied in sequence.
After reading the first line of input, sed applies all commands to it, then outputs the result. Then the second line of input is read in, all commands are applied to it... and this process is repeated. In the previous example, sed obtains input from the standard input device (i.e., the command interpreter, usually in the form of pipeline input). When one or more filenames are given as parameters on the command line, these files replace the standard input device as the input of sed. The output of sed will be sent to the standard output (display). Therefore:

cat filename | sed '10q' # Use pipeline input
sed '10q' filename # The same effect, but without using pipeline input
sed '10q' filename > newfile # Redirect the output to disk

To understand the usage instructions of sed commands, including how to use these commands through script files (instead of from the command line), please refer to "sed & awk" second edition, authors Dale Dougherty and Arnold Robbins (O'Reilly, 1997; http://www.ora.com), "UNIX Text Processing", authors
Dale Dougherty and Tim O'Reilly (Hayden Books, 1987) or the tutorial written by Mike Arst - the compressed package is named "U-SEDIT2.ZIP" (can be found on many sites). To explore the potential of sed, one must have sufficient understanding of "regular expressions". Information about regular expressions can be found in "Mastering Regular Expressions" by Jeffrey Friedl (O'reilly 1997).
The manual pages ("man") provided by the Unix system will also be helpful (try these commands "man sed", "man regexp", or see the part about regular expressions in "man ed"), but
the information provided by the manual is relatively "abstract" - which is also what it has been criticized for. However, it is not a textbook for teaching beginners how to use sed or regular expressions, but just a text reference for those who are familiar with these tools.

Bracket syntax: The previous examples basically use single quotes ('...') instead of double quotes ("...") for sed commands because sed is usually used on Unix platforms. Under single quotes, the Unix shell (command interpreter) will not interpret and execute the dollar sign ($) and backquote (`...`). Under double quotes, the dollar sign will be expanded to the value of a variable or parameter, and the command in the backquote will be executed and replaced with the output result. In "csh" and its derived shells, when using exclamation points (!), a backslash (\) for escaping must be added in front of it (like this: \!) to ensure that the above examples can run normally (including under the condition of using single quotes). The DOS version of Sed always uses double quotes ("...") instead of quotes to enclose commands.

Usage of '\t': To keep this document concise, we use '\t' to represent a tab character in the script. However, most versions of sed currently do not recognize the shorthand form of '\t', so when entering the tab character in the command line for the script, you should directly press the TAB key to enter the tab character instead of entering '\t'. The following tools support '\t' as a regular expression character to represent a tab character: awk, perl, HHsed, sedmod, and GNU sed v3.02.80.

Different versions of SED: There are some differences between different versions of sed, and it can be imagined that there will be differences in syntax between them. Specifically, most of them do not support using labels (:name) or branch commands (b, t) in the middle of editing commands, unless they are placed at the end. In this document, we try to use more portable syntax as much as possible so that users of most versions of sed can use these scripts. However, the GNU version of sed allows using more concise syntax. Imagine the mood of the reader when seeing a very long command:

sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d

The good news is that GNU sed can make the command more compact:

sed '/AAA/b;/BBB/b;/CCC/b;d' # Can even be written as
sed '/AAA\|BBB\|CCC/b;d'

In addition, please note that although many versions of sed accept commands like "/one/ s/RE1/RE2/" with a space before 's', some of these versions do not accept such commands: "/one/! s/RE1/RE2/". At this time, you only need to remove the space in the middle.

Speed optimization: When you need to improve the command execution speed for some reason (such as a large input file, slow processor or hard disk, etc.), you can consider adding an address expression in front of the substitution command ("s/.../.../") to improve the speed. For example:

sed 's/foo/bar/g' filename # Standard substitution command
sed '/foo/ s/foo/bar/g' filename # Faster speed
sed '/foo/ s//bar/g' filename # Shorthand form

When you only need to display the front part of the file or need to delete the content at the back, you can use the "q" command (exit command) in the script. When processing large files, this will save a lot of time. Therefore:

sed -n '45,50p' filename # Display lines 45 to 50
sed -n '51q;45,50p' filename # The same, but much faster

If you have other one-line scripts to share or you find an error in this document, please send an email to the author of this document (Eric Pement). Please remember to provide the version of sed you are using, the operating system on which the sed runs, and an appropriate description of the problem in the email. The one-line scripts referred to in this document refer to sed scripts with a command line length of 65 characters or less . The various scripts in this document were written or provided by the following authors:

Al Aab # Established the "seders" mailing list
Edgar Allen # Many aspects
Yiorgos Adamopoulos # Many aspects
Dale Dougherty # Author of "sed & awk"
Carlos Duarte # Author of "do it with sed"
Eric Pement # Author of this document
Ken Pizzini # Author of GNU sed v3.02
S.G. Ravenhall # Script to remove html tags
Greg Ubben # Made many contributions and provided a lot of help
-------------------------------------------------------------------------

Note 1: In most cases, sed scripts can be written in a single line regardless of their length (through the `-e' option and `;' sign) - as long as the command interpreter supports it, so the one-line scripts mentioned here are not only able to be written in one line but also have a length limit. Because the meaning of these one-line scripts does not lie in their being in one line. But making it convenient for users to use these compact scripts on the command line is their meaning.