Garbled text when copying a web page into Notepad - China DOS Union Forum

Test page:
http://bbs.xunshang.net/simple/index.php?t53815.html
For everyone's convenience in testing, please start cmd, then under the desktop path copy the following code:

wget http://bbs.xunshang.net/simple/index.php?t53815.html -O test.htm
htox32c /ip test.htm test.txt
write test.txt

I'm not familiar with html tags, so I converted it to txt

In the test.txt file on the desktop, there is garbled text at the end of every line (please open it with Write), how can you filter out this garbled text?

Third-party tools:
wget htox32c perl sed grep are all in my netdisk (the green one in my signature)

[ Last edited by plp626 on 2010-4-25 at 23:01 ]

Right-click on that web page, view source, flip forward two pages, and you'll know the reason.

It can be cleaned out.

What a disgusting forum...

This is garbled text added by the forum to prevent copying.

Implemented with www.topfisher.com/
topfisher is very powerful for web data collection and data analysis. Although it doesn't support image verification or user login verification, handling ordinary pages is a piece of cake. Among all the scraping software I've used so far, this one is the most flexible to implement based on code. The only stronger one is that Dedecms one, which can do real-time online collection.

//TopFisher v 2.10 script code
//Configuration section, you can directly modify configuration parameters here
InitLink("http://bbs.xunshang.net/simple/index.php?t53815.html")
DbConnStr("Provider=Microsoft.JET.OLEDB.4.0;Data source=data.mdb;Jet OLEDB:Engine Type=5;")
dbTable("myTable",5)

//-----------------------------------------------------------

//Code section
function main
webmem(DELETE,"<span style="display:none">","<br/>")
webmem(REPLACE,"<br/>","")
//This is the main function of the TPF script, please write the code here
_$str1=html.head.body.table.tr.td.table.tr.td.text
dellabel(_$str1)
printf(_$str1)

end


2010/04/26 02:06:39: main
2010/04/26 02:06:39: Creating .MDB file...
2010/04/26 02:06:39: Create .MDB file ok.
2010/04/26 02:06:39: open db ok.
2010/04/26 02:06:39: http://bbs.xunshang.net/simple/index.php?t53815.html
2010/04/26 02:06:39: write temp web file...
2010/04/26 02:06:39: webfile connect ok.
2010/04/26 02:06:39: Reading web file...
2010/04/26 02:06:39: Web file length is: 25798 bytes.
2010/04/26 02:06:39: building webBuffer...
2010/04/26 02:06:39: _$str1 = 
2010/04/26 02:06:39: database closed.
2010/04/26 02:06:39: all .tpf file finished.

webMem(p1(operation instruction)，p2(string parameter 1),? p3(string parameter 2)): directly filters the web page source
Explanation:
Directly performs string filtering on the downloaded web page source, for example: replacing certain strings, or deleting certain strings, etc. This function generally does not need to be used. Only when the source of certain web pages is rather messy and direct collection is troublesome, and some useless elements need to be filtered out first, is there a need to use the webMem function.

[ Last edited by dato on 2010-4-26 at 02:30 ]



wget http://bbs.xunshang.net/simple/index.php?t53815.html -O - | htox32c /IP  | sed "s/ .*//g" | sed "/^\+/d"

Here means non-Chinese characters.

Originally posted by asnahu at 2010-4-26 09:32 AM:
wget http://bbs.xunshang.net/simple/index.php?t53815.html -O - | htox32c /IP | sed "s/ .*//g" | sed "/^\+/d"

Here means non-Chinese characters.

I like this elegant code
=========================

But filters out the garbled characters on the web page, and also filters out useful data in the article, such as Arabic numerals and some English words

[ Last edited by plp626 on 2010-4-26 at 13:16 ]

Originally posted by dato at 2010-4-26 02:29 AM:
Implemented with http://www.topfisher.com/.
topfisher is very powerful for web data collection, data analysis. Although it doesn't support image verification, user login verification, but handling ordinary pages is a piece of cake. Thi ...

It is very powerful. I'm out of credits now, I'll add them next time

In Microsoft Office FrontPage, Find — HTML tag: set the tag to find to span, then:
1, you can directly set “Replace action” to “Replace contents only” and directly wipe out all the garbled text inside <span style="display:none">**********</span>...
2, you can directly set “Replace action” to “Replace tag and contents”, so even the span tag is removed and the world is clean again...

For the above purpose, you can also use sed to do cross-line matching on <span style="display:none">**********</span>, etc.

Floor1　plp626	Posted 2010-04-25 22:50
银牌会员　Posts 1,020　Credits 2,278
Test page: http://bbs.xunshang.net/simple/index.php?t53815.html For everyone's convenience in testing, please start cmd, then under the desktop path copy the following code: wget http://bbs.xunshang.net/simple/index.php?t53815.html -O test.htm htox32c /ip test.htm test.txt write test.txt I'm not familiar with html tags, so I converted it to txt In the test.txt file on the desktop, there is garbled text at the end of every line (please open it with Write), how can you filter out this garbled text? Third-party tools: wget htox32c perl sed grep are all in my netdisk (the green one in my signature) [ Last edited by plp626 on 2010-4-25 at 23:01 ]

Floor2　exzzz	Posted 2010-04-25 23:18
初级用户　Posts 167　Credits 194
Right-click on that web page, view source, flip forward two pages, and you'll know the reason. It can be cleaned out.

Floor3　Pierre	Posted 2010-04-26 00:11
初级用户　Posts 19　Credits 30
What a disgusting forum...

Floor4　Pierre	Posted 2010-04-26 00:12
初级用户　Posts 19　Credits 30
This is garbled text added by the forum to prevent copying.

Floor5　dato	Posted 2010-04-26 02:29
高级用户　Posts 377　Credits 916
Implemented with www.topfisher.com/ topfisher is very powerful for web data collection and data analysis. Although it doesn't support image verification or user login verification, handling ordinary pages is a piece of cake. Among all the scraping software I've used so far, this one is the most flexible to implement based on code. The only stronger one is that Dedecms one, which can do real-time online collection. //TopFisher v 2.10 script code //Configuration section, you can directly modify configuration parameters here InitLink("http://bbs.xunshang.net/simple/index.php?t53815.html") DbConnStr("Provider=Microsoft.JET.OLEDB.4.0;Data source=data.mdb;Jet OLEDB:Engine Type=5;") dbTable("myTable",5) //----------------------------------------------------------- //Code section function main webmem(DELETE,"<span style="display:none">","<br/>") webmem(REPLACE,"<br/>","") //This is the main function of the TPF script, please write the code here _$str1=html.head.body.table.tr.td.table.tr.td.text dellabel(_$str1) printf(_$str1) end 2010/04/26 02:06:39: main 2010/04/26 02:06:39: Creating .MDB file... 2010/04/26 02:06:39: Create .MDB file ok. 2010/04/26 02:06:39: open db ok. 2010/04/26 02:06:39: http://bbs.xunshang.net/simple/index.php?t53815.html 2010/04/26 02:06:39: write temp web file... 2010/04/26 02:06:39: webfile connect ok. 2010/04/26 02:06:39: Reading web file... 2010/04/26 02:06:39: Web file length is: 25798 bytes. 2010/04/26 02:06:39: building webBuffer... 2010/04/26 02:06:39: _$str1 = 2010/04/26 02:06:39: database closed. 2010/04/26 02:06:39: all .tpf file finished. webMem(p1(operation instruction)，p2(string parameter 1),? p3(string parameter 2)): directly filters the web page source Explanation: Directly performs string filtering on the downloaded web page source, for example: replacing certain strings, or deleting certain strings, etc. This function generally does not need to be used. Only when the source of certain web pages is rather messy and direct collection is troublesome, and some useless elements need to be filtered out first, is there a need to use the webMem function. [ Last edited by dato on 2010-4-26 at 02:30 ]

Floor6　asnahu	Posted 2010-04-26 09:32
初级用户　Posts 53　Credits 99
wget http://bbs.xunshang.net/simple/index.php?t53815.html -O - \| htox32c /IP \| sed "s/ .*//g" \| sed "/^\+/d" Here means non-Chinese characters.

Floor7　plp626	Posted 2010-04-26 13:13
银牌会员　Posts 1,020　Credits 2,278
Originally posted by asnahu at 2010-4-26 09:32 AM: wget http://bbs.xunshang.net/simple/index.php?t53815.html -O - \| htox32c /IP \| sed "s/ .//g" \| sed "/^\+/d" Here means non-Chinese characters. I like this elegant code ========================= But filters out the garbled characters on the web page, and also filters out useful data in the article, such as Arabic numerals and some English words [ Last edited by plp626 on 2010-4-26 at 13:16* ]

Floor8　plp626	Posted 2010-04-26 13:19
银牌会员　Posts 1,020　Credits 2,278
Originally posted by dato at 2010-4-26 02:29 AM: Implemented with http://www.topfisher.com/. topfisher is very powerful for web data collection, data analysis. Although it doesn't support image verification, user login verification, but handling ordinary pages is a piece of cake. Thi ... It is very powerful. I'm out of credits now, I'll add them next time

Floor9　Hanyeguxing	Posted 2010-04-26 14:09
银牌会员　Posts 897　Credits 1,039　From 在地狱中仰望天堂
In Microsoft Office FrontPage, Find — HTML tag: set the tag to find to span, then: 1, you can directly set “Replace action” to “Replace contents only” and directly wipe out all the garbled text inside <span style="display:none">********</span>... 2, you can directly set “Replace action” to “Replace tag and contents”, so even the span tag is removed and the world is clean again... For the above purpose, you can also use sed to do cross-line matching on <span style="display:none">********</span>, etc.

Floor10　fsfss	Posted 2010-04-26 23:32
初级用户　Posts 20　Credits 22