China DOS Union

Test page:
http://bbs.xunshang.net/simple/index.php?t53815.html
For everyone's convenience in testing, please start cmd, then under the desktop path copy the following code:

wget http://bbs.xunshang.net/simple/index.php?t53815.html -O test.htm

htox32c /ip test.htm test.txt

write test.txt

I'm not familiar with html tags, so I converted it to txt

In the test.txt file on the desktop, there is garbled text at the end of every line (please open it with Write), how can you filter out this garbled text?

Third-party tools:
wget htox32c perl sed grep are all in my netdisk (the green one in my signature)

[ Last edited by plp626 on 2010-4-25 at 23:01 ]

Right-click on that web page, view source, flip forward two pages, and you'll know the reason.

It can be cleaned out.

What a disgusting forum...

This is garbled text added by the forum to prevent copying.

Implemented with www.topfisher.com/
topfisher is very powerful for web data collection and data analysis. Although it doesn't support image verification or user login verification, handling ordinary pages is a piece of cake. Among all the scraping software I've used so far, this one is the most flexible to implement based on code. The only stronger one is that Dedecms one, which can do real-time online collection.

//TopFisher v 2.10 script code

//Configuration section, you can directly modify configuration parameters here

InitLink("http://bbs.xunshang.net/simple/index.php?t53815.html")

DbConnStr("Provider=Microsoft.JET.OLEDB.4.0;Data source=data.mdb;Jet OLEDB:Engine Type=5;")

dbTable("myTable",5)



//-----------------------------------------------------------



//Code section

function main

webmem(DELETE,"<span style="display:none">","<br/>")

webmem(REPLACE,"<br/>","")

//This is the main function of the TPF script, please write the code here

_$str1=html.head.body.table.tr.td.table.tr.td.text

dellabel(_$str1)

printf(_$str1)



end

2010/04/26 02:06:39: main

2010/04/26 02:06:39: Creating .MDB file...

2010/04/26 02:06:39: Create .MDB file ok.

2010/04/26 02:06:39: open db ok.

2010/04/26 02:06:39: http://bbs.xunshang.net/simple/index.php?t53815.html

2010/04/26 02:06:39: write temp web file...

2010/04/26 02:06:39: webfile connect ok.

2010/04/26 02:06:39: Reading web file...

2010/04/26 02:06:39: Web file length is: 25798 bytes.

2010/04/26 02:06:39: building webBuffer...

2010/04/26 02:06:39: _$str1 = 



2010/04/26 02:06:39: database closed.

2010/04/26 02:06:39: all .tpf file finished.

webMem(p1(operation instruction)，p2(string parameter 1),? p3(string parameter 2)): directly filters the web page source
Explanation:
Directly performs string filtering on the downloaded web page source, for example: replacing certain strings, or deleting certain strings, etc. This function generally does not need to be used. Only when the source of certain web pages is rather messy and direct collection is troublesome, and some useless elements need to be filtered out first, is there a need to use the webMem function.

[ Last edited by dato on 2010-4-26 at 02:30 ]

wget http://bbs.xunshang.net/simple/index.php?t53815.html -O - | htox32c /IP  | sed "s/ .*//g" | sed "/^\+/d"

Here means non-Chinese characters.

Originally posted by asnahu at 2010-4-26 09:32 AM:

wget http://bbs.xunshang.net/simple/index.php?t53815.html -O - | htox32c /IP  | sed "s/ .*//g" | sed "/^\+/d"

Here means non-Chinese characters.

I like this elegant code
=========================

But filters out the garbled characters on the web page, and also filters out useful data in the article, such as Arabic numerals and some English words

[ Last edited by plp626 on 2010-4-26 at 13:16 ]

Originally posted by dato at 2010-4-26 02:29 AM:
Implemented with http://www.topfisher.com/.
topfisher is very powerful for web data collection, data analysis. Although it doesn't support image verification, user login verification, but handling ordinary pages is a piece of cake. Thi ...

It is very powerful. I'm out of credits now, I'll add them next time

In Microsoft Office FrontPage, Find — HTML tag: set the tag to find to span, then:
1, you can directly set “Replace action” to “Replace contents only” and directly wipe out all the garbled text inside <span style="display:none">**********</span>...
2, you can directly set “Replace action” to “Replace tag and contents”, so even the span tag is removed and the world is clean again...

For the above purpose, you can also use sed to do cross-line matching on <span style="display:none">**********</span>, etc.