China DOS Union

(Linux Vim Editing of Binary and Hexadecimal Files)
http://cloudtech.iteye.com/blog/1692452

(Translated) vim+xxd=Powerful Hexadecimal Editor

Original address: http://xineohpanihc.iteye.com/blog/1148741
Vim configuration file address: https://digital-ldentification.googlecode.com/files/vimrc
Vim is a powerful editor in hacker culture. By using it to call the external hexadecimal file display command xxd, you can smoothly edit binary files.
Among them, % represents the path of the current file in vim Command Line, xxd displays a file in hexadecimal, and xxd -r> restores hexadecimal to binary.
Here is an example to back up our MBR.
sudo dd if=/dev/sda of=~/mbr.backup bs=512 count=1
By the way: It contains the partition table. At this time, we can use this command to restore our MBr:
sudo dd if=~/mbr.backup of=/dev/sda bs=512 count=1
sudo dd if=~/mbr.backup of=/dev/sda bs=1 count=66 seek=446 (restore partition table)
Well, this is our MBR file, which is binary.

First open it in binary mode:
vim mbr.backup -b
Then enter in vim command mode:
:%!xxd #The parameter % refers to the currently edited file
At this time, the MBR file will be displayed in hexadecimal, and now you can edit it..
========================================================
0000090: be88 7de8 1c01 be05 7cf6 c280 7448 b441 ..}.....|...tH.A
00000a0: bbaa 55cd 135a 5272 3d81 fb55 aa75 3783 ..U..ZRr=..U.u7.
00000b0: e101 7432 31c0 8944 0440 8844 ff89 4402 ..t21..D.@.D..D.
00000c0: c704 1000 668b 1e5c 7c66 895c 0866 8b1e ....f..\|f.\.f..
00000d0: 607c 6689 5c0c c744 0600 70b4 42cd 1372 `|f.\..D..p.B..r
00000e0: 05bb 0070 eb76 b408 cd13 730d f6c2 800f ...p.v....s.....
00000f0: 84d0 00be 937d e982 0066 0fb6 c688 64ff .....}...f....d.
0000100: 4066 8944 040f b6d1 c1e2 0288 e888 f440 @f.D...........@
0000110: 8944 080f b6c2 c0e8 0266 8904 66a1 607c .D.......f..f.`|
0000120: 6609 c075 4e66 a15c 7c66 31d2 66f7 3488 f..uNf.\|f1.f.4.
0000130: d131 d266 f774 043b 4408 7d37 fec1 88c5 .1.f.t.;D.}7....
0000140: 30c0 c1e8 0208 c188 d05a 88c6 bb00 708e 0........Z....p.
0000150: c331 dbb8 0102 cd13 721e 8cc3 601e b900 .1......r...`...
0000160: 018e db31 f6bf 0080 8ec6 fcf3 a51f 61ff ...1..........a.
0000170: 265a 7cbe 8e7d eb03 be9d 7de8 3400 bea2 &Z|..}....}.4...
0000180: 7de8 2e00 cd18 ebfe 4752 5542 2000 4765 }.......GRUB .Ge
0000190: 6f6d 0048 6172 6420 4469 736b 0052 6561 om.Hard Disk.Rea
00001a0: 6400 2045 7272 6f72 0d0a 00bb 0100 b40e d. Error........
00001b0: cd10 ac3c 0075 f4c3 e3b9 76fb 0000 8001 ...<.u....v.....
00001c0: 0100 07fe ffff 3f00 0000 499d d801 00fe ......?...I.....
00001d0: ffff 0ffe ffff c59d d801 eb4c 6a23 0000 ...........Lj#..
00001e0: 0000 0000 0000 0000 0000 0000 0000 0000 ................
00001f0: 0000 0000 0000 0000 0000 0000 0000 55aa ..............U.

================================================================
After editing, we need to restore it to a binary file:
:%!xxd -r #The parameter -r means to convert the current hexadecimal to binary

Okay, save and exit:
:wq
==========================================
VIM Hexadecimal Display Error Source: http://www.cnblogs.com/lidp/archive/2009/06/26/1697873.html

When opening a jpg file with vim under Linux and using %!xxd for hexadecimal display, the file header is displayed as "3f3f
3f3f 0011 0804" and the file end is displayed as "3f3f
0a"; while the same operation under Windows displays "ffd8 ffc0 0011 0804" and "ffd9
0a", which is the correct jpeg file header and file end mark.
Very strange!
At first, I thought that jpeg had different file headers under Windows and Linux. Later, I removed the jpg suffix and it was just a pure file, and the phenomenon was still the same. I considered that it might not be the difference of the operating system.
Reopen the jpeg data with ghex and find that it is displayed normally as "ffd8 ffc0 0011
0804" and "ffd9 0a", which is correct.
It is estimated that it should be a problem with vim.
The ASCII code of 3f is?, which means that vim did not parse the file header and end normally. Is it related to the encoding format used by vim to parse the file?
Open the .vimrc configuration item and block the following sentence:
set fileencodings=utf-8,gb2312,gbk,gb18030,ucs-bom
Then open the jpeg file with vim, and it is displayed as "ffd8 ffc0 0011 0804" and "ffd9 0a",
Display correctly.
Originally, in order to support recognizing and displaying Chinese, I specified the fileencodings of vim,
When vim opens the file, it will use the specified encoding format to parse the data. Unfortunately, the file header FFD8 and tail FFD9 of jpeg are not the encoding of any Chinese. Vim cannot find the corresponding Chinese character, so it is displayed as??, that is: 3f3f.
At this point, all the confusion is cleared.




Generally speaking, files are divided into two categories: text and binary files. Text file editing is very easy. Notepad and UltraEdit under Windows are very easy to use. There are also many tools for binary file editing under Windows. UltraEdit is also good. But under Linux? Today I finally found a good method, but the final source cannot be found.
First create a binary file:
view sourceprint?

Ruby code Collection code
$echo-n"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz">test.bin
$cattest.bin
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
$
Note that the echo must be followed by the -n option, otherwise a newline will be automatically added. Then open test.bin with vim
Ruby code Collection code
$vim-btest.bin
The -b option of vim tells vim that the opened file is a binary file. If not specified, a 0x0a, that is, a newline character, will be added at the end.
In command mode, type:
Ruby code Collection code
:%!xxd

If vim is not followed by the -b option, the hateful 0x0a will appear:

If there is a -b option, there will be no such situation:

Then enter edit mode to change. Just change. I changed 41 and 42 corresponding to A and B to 61 and 62, and changed 61 and 62 corresponding to a and b to 41 and 42.

Back to command mode and enter:
Ruby code Collection code
:%!xxd-r

At this time, you can find that the positions of AB and ab are swapped.

Finally, enter :wq in command mode to save and exit.


-----------------------------------------------------------------------------------------------------------


Vim can edit binary files very conveniently. Personally, I think it is better than the binary editing method of emacs. The editing of binary files in vim is to first dump the file into its binary text form through the external program xxd, and then you can edit the file in the usual editing method. After editing, you can convert it back to the original form with xxd.
It can be carried out in the following steps:
(1) First edit this file in binary mode: vim-b datafile
(2) Now use xxd to convert this file to hexadecimal: :%!xxd
The text looks like this:
0000000: 1f8b 0808 39d7 173b 0203 7474 002b 4e49 ....9..;..tt.+NI
0000010: 4b2c 8660 eb9c ecac c462 eb94 345e 2e30 K,.`.....b..4^.0
0000020: 373b 2731 0b22 0ca6 c1a2 d669 1035 39d9 7;'1.".....i.59.
Now you can read and edit these texts as you like. Vim treats this information as plain text. Changing the hexadecimal part does not cause a change in the displayable character part, and vice versa.
(3) Finally, use the following command to convert it back: :%!xxd-r
Only the changes in the hexadecimal part will be adopted. The changes in the right displayable text part are ignored.
xxd is a command of linux. Vim can call external commands through "!", and its function is to perform hexadecimal dump or vice versa.
VIM Editing Binary File (Vim Manual Record)
*23.4* Binary File
You can use Vim to edit binary files. Vim is not designed for this, so there are several limitations. But you can read a file, change a character, and then save it. The result is that only that one character in your file is changed, and the rest are exactly the same as the original one.
To ensure that Vim does not use its smart tricks in the wrong place, start Vim with the "-b" parameter:
vim -b datafile
This parameter sets the 'binary' option. Its function is to exclude all unexpected side effects. For example, 'textwidth' is set to zero to avoid the text line being arbitrarily typeset. Also, the file is read in Unix file format.
The binary mode can be used to modify the message message of a certain program. Be careful not to insert or delete any characters, which will cause the program to run incorrectly. Enter replacement mode with the "R" command.
Many characters in the file are not displayable. Use Hex format to display their values:
:set display=uhex
In addition, you can also use the command "ga" to display the value of the character under the cursor. When the cursor is on an <Esc> character, the output of this command looks like this:
<^


















































]
xxd -r ]

xxd can create a hexdump of a given file or standard input, and can also restore a hexdump to the previous binary file. If no file is specified or the file name is specified as -, standard input is used. If no output file is specified or the file name is specified as -, standard output is used.

-a
toggle autoskip: A single '*' replaces nul-lines. Default off.

-b
Use binary display instead of the default hexadecimal

xxd -l 32 -b scz.sh
0000000: 00100011 00100001 00100000 00101111 01100010 01101001 #! /bi
0000006: 01101110 00101111 01110011 01101000 00001010 00100011 n/sh.#
000000c: 00001010 00100011 00100000 11001000 10100001 11010110 .# ...
0000012: 11110111 11001001 11101000 10110001 10111000 10111010 ......
0000018: 11000101 00001010 00100011 00100000 01100011 01100001 ..# ca
000001e: 01110100 00100000 t

-c cols
Default 16, when -i is specified, 12 is used, when -p is specified, 30 is used, when -b is specified, 6 is used, maximum 256
-c8, -c 8, -c 010 are equivalent, and other options are similar.

xxd -c 8 -l 32 -g 1 scz.sh
0000000: 23 21 20 2f 62 69 6e 2f #! /bin/
0000008: 73 68 0a 23 0a 23 20 c8 sh.#.# .
0000010: a1 d6 f7 c9 e8 b1 b8 ba ........
0000018: c5 0a 23 20 63 61 74 20 ..# cat

-E
Do not display in ASCII, but use EBCDIC encoding, which does not affect hexadecimal display

xxd -E -l 32 -g 1 scz.sh
0000000: 23 21 20 2f 62 69 6e 2f 73 68 0a 23 0a 23 20 c8 ......>........H
0000010: a1 d6 f7 c9 e8 b1 b8 ba c5 0a 23 20 63 61 74 20 .O7IY...E..../..

-g bytes
Compare the following three display methods, the default bytes is 2, and 1 is used when -b is specified.

xxd -l 32 -g 0 scz.sh
0000000: 2321202f62696e2f73680a230a2320c8 #! /bin/sh.#.# .
0000010: a1d6f7c9e8b1b8bac50a232063617420 ..........# cat

xxd -l 32 -g 1 scz.sh
0000000: 23 21 20 2f 62 69 6e 2f 73 68 0a 23 0a 23 20 c8 #! /bin/sh.#.# .
0000010: a1 d6 f7 c9 e8 b1 b8 ba c5 0a 23 20 63 61 74 20 ..........# cat

xxd -l 32 -g 2 scz.sh
0000000: 2321 202f 6269 6e2f 7368 0a23 0a23 20c8 #! /bin/sh.#.# .
0000010: a1d6 f7c9 e8b1 b8ba c50a 2320 6361 7420 ..........# cat

-h
Display help information

-i
Display in C language array form. If the input comes from stdin, there will be differences

unsigned char scz_sh = {
0x23, 0x21, 0x20, 0x2f, 0x62, 0x69, 0x6e, 0x2f, 0x73, 0x68, 0x0a, 0x23,
0x0a, 0x23, 0x20, 0xc8, 0xa1, 0xd6, 0xf7, 0xc9, 0xe8, 0xb1, 0xb8, 0xba,
0xc5, 0x0a, 0x23, 0x20, 0x63, 0x61, 0x74, 0x20
};
unsigned int scz_sh_len = 32;

Note the difference between the execution effects of the following two commands

xxd -l 12 -i < scz.sh
0x23, 0x21, 0x20, 0x2f, 0x62, 0x69, 0x6e, 0x2f, 0x73, 0x68, 0x0a, 0x23

xxd -l 12 -i scz.sh
unsigned char scz_sh = {
0x23, 0x21, 0x20, 0x2f, 0x62, 0x69, 0x6e, 0x2f, 0x73, 0x68, 0x0a, 0x23
};
unsigned int scz_sh_len = 12;

-l len
Only display <len> bytes

-p
Display in continuous hexadecimal representation. Compare the following different display effects

xxd -u -l 32 scz.sh
0000000: 2321 202F 6269 6E2F 7368 0A23 0A23 20C8 #! /bin/sh.#.# .
0000010: A1D6 F7C9 E8B1 B8BA C50A 2320 6361 7420 ..........# cat

xxd -u -l 32 -p scz.sh
2321202F62696E2F73680A230A2320C8A1D6F7C9E8B1B8BAC50A23206361
7420

-r
reverse operation: convert (or patch) hexdump into binary.
If not writing to stdout, xxd writes into its output file without
truncating it. Use the combination -r -p to read plain hexadecimal
dumps without line number information and without a particular
column layout. Additional Whitespace and line-breaks are allowed
anywhere.

This option must be tested in advance

-seek offset
When used after -r : revert with <offset> added to file positions
found in hexdump.

-s seek
start at <seek> bytes abs. (or rel.) infile offset. + indicates
that the seek is relative to the current stdin file position
(meaningless when not reading from stdin). - indicates that the seek
should be that many characters from the end of the input (or if
combined with + : before the current stdin file position).
Without -s option, xxd starts at the current file position.

-u
Use uppercase hexadecimal letters for display, default use lowercase hexadecimal letters

-v
Display version information

★ Precautions for Using xxd

xxd -r has some builtin magic while evaluating line number information. If the ouput file
is seekable, then the linenumbers at the start of each hexdump line may be out of order,
lines may be missing, or overlapping. In these cases xxd will lseek(2) to the next position.
If the output file is not seekable, only gaps are allowed, which will be filled by null-
bytes.

xxd -r never generates parse errors. Garbage is silently skipped.

When editing hexdumps, please note that xxd -r skips everything on the input line after
reading enough columns of hexadecimal data (see option -c). This also means, that changes to
the printable ascii (or ebcdic) columns are always ignored. Reverting a plain (or
postscript) style hexdump with xxd -r -p does not depend on the correct number of columns.
Here an thing that looks like a pair of hex-digits is interpreted.

xxd -s +seek may be different from xxd -s seek , as lseek(2) is used to "rewind" input. A
'+' makes a difference if the input source is stdin, and if stdin's file position is not at
the start of the file by the time xxd is started and given its input. The following
examples may help to clarify (or further confuse!)...

Rewind stdin before reading; needed because the `cat' has already read to the end of stdin.
% sh -c 'cat > plain_copy; xxd -s 0 > hex_copy' < file

Hexdump from file position 0x480 (=1024+128) onwards. The `+' sign means "relative to the
current position", thus the `128' adds to the 1k where dd left off.
% sh -c 'dd of=plain_snippet bs=1k count=1; xxd -s +128 > hex_snippet' < file

Hexdump from file position 0x100 ( = 1024-768) on.
% sh -c 'dd of=plain_snippet bs=1k count=1; xxd -s +-768 > hex_snippet' < file

However, this is a rare situation and the use of `+' is rarely needed. the author prefers
to monitor the effect of xxd with strace(1) or truss(1), whenever -s is used.

★ xxd Usage Examples

.. Display starting from offset 0x10, that is, the first line is not displayed by default
xxd -s 0x10 scz.sh

.. Display the last 0x30 bytes (3 lines by default)
xxd -s -0x30 -g 1 scz.sh

.. Display 120 bytes, display continuously, 20 bytes per line
xxd -l 120 -p -c 20 scz.sh

.. Display 120 bytes, 12 bytes per line
xxd -l 120 -c 12 -g 1 scz.sh

.. Display 12 bytes starting from offset 0x6c
xxd -l 12 -c 12 -g 1 -s 0x6c scz.sh

.. Create a file of 65537 bytes in size, with all bytes 0x00 except the last byte is 'A'
echo 10000: 41 | xxd -r > scz.txt

od -A x -t x1 scz.txt
000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
010000 41
010001

.. Use autoskip option

xxd -a scz.txt
0000000: 0000 0000 0000 0000 0000 0000 0000 0000 ................
*
0010000: 41 A

Note that the autoskip function is off by default, but od seems to be on by default.

.. Create a file containing only one 'A' byte
echo '010000: 41' | xxd -r -s -0x10000 > scz_1

Note to compare these effects
echo '010: 41 42 43 44 45 46' | xxd -r -s 0 > scz_1

echo '010: 41 42 43 44 45 46' | xxd -r -s -0x10 > scz_1
Equivalent to
echo '0: 41 42 43 44 45 46' | xxd -r -s 0 > scz_1

echo '010: 41 42 43 44 45 46' | xxd -r -s -0x12 > scz_1
An error is reported when executing, xxd: sorry, cannot seek backwards.

echo '010: 41 42 43 44 45 46' | xxd -r -s 0x10 > scz_1
Equivalent to
echo '020: 41 42 43 44 45 46' | xxd -r -s 0 > scz_1

.. Copy the input file to the output file and add 100 bytes of 0x00 to the front of the output file
xxd scz_1 | xxd -r -s 100 > scz_2

od -A x -t x1 scz_2
000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
*
000060 00 00 00 00 41
000065

.. Binary file patching
echo '0: 41 42 43 44 45 46' | xxd -r -s 0 > scz_1
od -A x -t x1 scz_1
echo '4: 47 48' | xxd -r - scz_1
od -A x -t x1 scz_1

.. Use xxd as a filter within an editor such as vim(1) to hexdump a region
marked between `a' and `z'.
:'a,'z!xxd

.. Use xxd as a filter within an editor such as vim(1) to recover a
binary hexdump marked between `a' and `z'.
:'a,'z!xxd -r

.. Use xxd as a filter within an editor such as vim(1) to recover one line
of a hexdump. Move the cursor over the line and type:
!!xxd -r

.. Read single characters from a serial line
xxd -c1 < /dev/term/b &
stty < /dev/term/b -echo -opost -isig -icanon min 1
echo -n foo > /dev/term/b

[ Last edited by zzz19760225 on 2016-12-15 at 01:31 ]