China DOS Union

Because in my actual use I needed UTF8-to-GBK encoding conversion, I wrote one with GAWK. In fact, I had already been using it in an earlier post. This time I整理ed it a bit, and it now supports bidirectional encoding conversion. I made a complete GBK-to-UTF8 conversion mapping table myself. During the process I found that there are quite a few differences between the system's conversion results and iconv's conversion results, and the mapping table uses the former.
This script supports encoding conversion through pipes and files, with the result output to the screen. There are not many parameters supported yet, but it does have parameter integrity checking and can handle unordered calls with multiple parameters. It uses a new script release method: when the source file is modified, the script will be updated automatically. There are also error messages and a dependency-file integrity checking mechanism. I hope these little tricks will be helpful to everyone in writing batch files.
GAWK download link: http://www.klabaster.com/progs/gawk32.zip
The script and conversion table are in the attachment.

1. :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
   
2. :: gbk2utf8.cmd -V0.1 -- GBK & UTF8 encoding conversion
   
3. :: 无奈何@cn-dos.net - 2006-11-28 - CMD & GAWK
   
4. :: Usage: gbk2utf8 /I file...
   
5. :: Supported files: - gawk.exe gbk2utf8.dat
   
6. :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
   
7. @echo off
   
8. setlocal
   
9. set self="%~f0"
   
10. set AwkScript="%temp%\%~n0%~z0.awk"
    
11. set path=%path%;%~dp0;%cd%
    
12. set nofile=
    
13. set error=
    
14. set input=
    
15. 
    
16. ::Dependency file integrity check
    
17. for %%i in (gawk.exe gbk2utf8.dat) do (
    
18. @if "%%~$PATH:i" == "" (
    
19. echo.The required dependency file "%%i" is missing.
    
20. set nofile=1
    
21. ) else ( set %%~ni="%%~$PATH:i" )
    
22. )
    
23. if defined nofile goto :EOF
    
24. ::Update script after file changes
    
25. if not exist %AwkScript% (
    
26. del /q "%temp%\%~n0*.awk" 2>nul
    
27. gawk "/^#<-1/,/^#>-1/{if(!/^#/)print}" %self% >%AwkScript%
    
28. )
    
29. 
    
30. :ParseLoop
    
31. if "%~1" == "" goto Start
    
32. if "%~1" == "?" goto SwitchH
    
33. if "%~1" == "/?" goto SwitchH
    
34. rem Process parameters and jump to the corresponding label.
    
35. for %%s in (U u I i h H) do if "%~1"=="/%%s" goto Switch%%s
    
36. if "%F_input%" == "1" (
    
37. if not exist "%~1" set error=Warning: file "%~1" does not exist. & goto error
    
38. set input=%input% "%~1"
    
39. shift
    
40. goto ParseLoop
    
41. )
    
42. if "%F_input%" == "-1" shift & goto ParseLoop
    
43. set error=Error: incorrect parameter format - "%1" !
    
44. goto error
    
45. 
    
46. :SwitchI
    
47. set F_input=1
    
48. if "%~2" == "-" set F_input=-1
    
49. shift
    
50. goto ParseLoop
    
51. 
    
52. :SwitchU
    
53. set F=-1
    
54. shift
    
55. goto ParseLoop
    
56. 
    
57. :error
    
58. echo.%error%
    
59. echo.
    
60. :SwitchH
    
61. echo.gbk2utf8 V0.1 -- GBK ^& UTF8 encoding conversion
    
62. echo.
    
63. echo.Usage: 1、%~n0
    
64. echo. 2、%~n0 /I file...
    
65. echo. 3、%~n0 /I -
    
66. echo.
    
67. echo.Options: /? displays this brief help, equivalent to /H .
    
68. echo. /U converts UTF8 to GBK; the default is GBK to UTF8.
    
69. echo. /I specifies the file to convert; “-” gets it from standard output.
    
70. echo. This parameter may be omitted; by default it will be obtained from standard output.
    
71. echo. When specifying a file to convert, the /I parameter cannot be omitted.
    
72. goto :EOF
    
73. 
    
74. :Start
    
75. if "%input%" == "" set F_input=-1
    
76. if "%F_input%" == "-1" (
    
77. gawk -v F=%F% -f %AwkScript%
    
78. ) else (
    
79. gawk -v F=%F% -f %AwkScript% %input%
    
80. )
    
81. goto :EOF
    
82. 
    
83. :AwkScript
    
84. #<-1
    
85. function gbk2utf8(string,flag, reg, gbkreg, utf8reg, char, result){
    
86. gbkreg="|"
    
87. utf8reg="||\xe0||\xf0|"
    
88. reg=gbkreg
    
89. if (flag==-1)
    
90. reg=utf8reg
    
91. RLENGTH = 1
    
92. while(RLENGTH != -1){
    
93. match(string,reg)
    
94. char=substr(string,RSTART,RLENGTH)
    
95. if (RLENGTH>1)
    
96. char=charset
    
97. result=result char
    
98. string=substr(string,RSTART+RLENGTH)
    
99. }
    
100. return result
     
101. }
     
102. 
     
103. BEGIN {
     
104. FS=","
     
105. if (!F) F=1
     
106. if (F==1) {
     
107. while((getline<"gbk2utf8.dat") > 0)
     
108. charset=$2
     
109. }
     
110. else{
     
111. while((getline<"gbk2utf8.dat") > 0)
     
112. charset=$1
     
113. }
     
114. close("gbk2utf8.dat")
     
115. }
     
116. {
     
117. x=gbk2utf8($0,F)
     
118. print x
     
119. }
     
120. #>-1
     
121. goto :EOF
     

Posted by 无奈何 2006-11-30 01:02

[ Last edited by 无奈何 on 2006-11-30 at 02:04 PM ]

Strong post, leaving a mark. (Brother electronixtar, please don't mind the infringement ^_^) Brother Wunaihe is a mountain I can never climb over!

Previously, I found a GBK-Unicode encoding comparison table (with more than 7,000 comparison characters) online, and I used batch processing to calculate and generate a GBK-UTF-8 encoding comparison table. As long as I use the dictionary lookup for method, I can instantly extract the UTF-8 encoding. Let me look for it...

Hey, I still don't understand that gawk part. I'm learning it.

So wonderful～～～

ccwan, you're being too modest. I'm not a pro either.
zxcv, brother, more than 7,000 are of GB2312, the complete GBK has over 20,000, with a total of 22,046 code points.

RE vkill
GAWK supports multi-byte encoding. The advantage of this processing method is that as long as you write a regular expression that matches the encoding, it can be generally used for conversion of other encodings. It should be said that it is relatively easy to understand. The only thing to do is to obtain the character length and intercept it.

Originally posted by 无奈何 at 2006-11-29 12:51:
Brother ccwan is being modest, and I'm not a expert either.
Brother zxcv, the 7k+ ones are of GB2312, the complete GBK has over 20k, with a total of 22046 code points.

Indeed, it seems I need to find a complete GBK to UTF-8 conversion table.

Have obtained the complete gbk2utf8 encoding library

Found that there are many different data between this gbk2utf8 encoding library and the GB version I converted:
For example:
gbk2utf8

A1A42C C2B7
A1A52C CB89
A1A62C CB87
A1A72C C2A8

The red part is the identification of gbk

gb2utf8

a1a4 E383BB
a1a5 E08B89
a1a6 E08B87
a1a7 E082A8

It seems that there are some incorrect code comparisons in the GB part of this gbk2utf8 encoding library, and no errors have been found in the codes beyond the library GB part for the time being

RE zxcv

My one has a separator "," added in the middle, which is 0X2C.

Here is a piece of AWK script I wrote to generate all GBK characters. Just save it as UTF8 to get a comparison table of the two encodings.

BEGIN{ #GBK/1: A1A1-A9FE Graphic symbol area GB 2312 non-Chinese character symbol area for (i=0xa1;i<=0xa9;i++) for (j=0xa1;j<=0xfe;j++) if (j!=0x7f) printf("%c%c\n",i,j) #GBK/2: B0A1-F7FE Chinese character area GB 2312 Chinese character area for (i=0xb0;i<=0xf7;i++) for (j=0xa1;j<=0xfe;j++) if (j!=0x7f) printf("%c%c\n",i,j) #GBK/3: 8140-A0FE Chinese character area for (i=0x81;i<=0xa0;i++) for (j=0x40;j<=0xfe;j++) if (j!=0x7f) printf("%c%c\n",i,j) #GBK/4: AA40-FEA0 Chinese character area for (i=0xaa;i<=0xfe;i++) for (j=0x40;j<=0xa0;j++) if (j!=0x7f) printf("%c%c\n",i,j) #GBK/5: A840-A9A0 Graphic symbol area for (i=0xa8;i<=0xa9;i++) for (j=0x40;j<=0xa0;j++) if (j!=0x7f) printf("%c%c\n",i,j) }

[ Last edited by 无奈何 on 2006-11-30 at 03:50 AM ]

Only the following codes in the GB part cannot be found corresponding ones

A8BB C991
A8BD C584
A8BE C588
A8C0 C9A1

First try it ^_^

I didn't understand it, so I'll give it a thumbs up first

Too profound. Learning.

Looks like C++

GAWK is really powerful～～～