Simple detection of whether a proxy server is available using wget - China DOS Union Forum

　　In daily internet surfing, sometimes proxy servers are used. The usual way is to go to some websites that provide proxy server lists to find the proxies we want. But there are many proxies above that are unavailable.难道要 manually test one by one which proxy is available? Our batch script can complete this task.

　　Using the proxy function of wget, download a fixed web page, such as www.baidu.com. I downloaded the web page www.ip138.com to achieve the purpose of simply detecting whether the proxy server lists of some websites are available. The website I used to provide the proxy server list is: http://www.cnproxy.com/proxyedu1.html

　　This batch processing cannot perfectly achieve our purpose. There are still many parameters and formats that can be modified. Everyone can modify the statements and parameters in it according to their own wishes. Everyone can also post their opinions if they have any. This batch processing is best run in a folder because many files will be generated in the middle.

　　Regarding the small tool wget, you can search in the forum to find it.

　　Here, I would like to thank Brother electronixtar for the proposal and Brother redtek for the help.
@echo off&setlocal ENABLEDELAYEDEXPANSION&title 代理检测 if exist proxyedu1.html del proxyedu1.html /a/s >nul if exist useproxy.txt del /a/q useproxy.txt echo 于 %date% %time% 检测 http://www.cnproxy.com/proxyedu1.html 可用代理如下: >useproxy.txt echo;>>useproxy.txt echo ############################################################# >>useproxy.txt wget http://www.cnproxy.com/proxyedu1.html >nul 2>nul for /f "tokens=1,2 delims=^<trd>: " %%a in ('type proxyedu1.html ^| findstr /c:"HTTP" ') do ( set set= set set=%%a set "set=!set:&#8205=!" set "set=!set!:%%b" echo 正在检测 !set! 代理，请稍等.... wget -q -T 3 -t 1 -e "http_proxy=!set!" http://www.ip138.com -O - >nul 2>nul && echo 检测到 ******* !set! ******* 代理可用&&echo ----!set! >>useproxy.txt ) start useproxy.txt goto :eof

[ Last edited by lxmxn on 2006-11-27 at 01:01 PM ]

Not bad, not bad. We should dig deeper into the functions of wget.

Here are a few points to supplement about wget:

wget -q does not output download record information

wget -O - outputs the downloaded content to the screen instead of writing to a file

　　Thanks for the moderator's guidance.

　　Since I just started learning to use wget, I haven't discovered so many useful parameters yet. Even the -t parameter was only found out after searching Baidu. Sweat.

　　I'll improve the code again.

Such a fun code, it's practical and provides more convenience for downloading~ :))) Appreciate it~~

I also tried wget before && didn't succeed, maybe I didn't add parameters!

Top it's a boon for users of our education network!

Forgot to add points, original work is hard, highly recommend

The automatic proxy setup done 2 months ago was a pity because there was no way to verify it. Now heh~ thanks

What a good thing. Will you be preparing to add a sorting function next time?

When joining an automatic generation of pac configuration file ~~ haha

GNU Wget 1.10.2, a non-interactive network retriever.
Usage: wget ... ... Mandatory arguments to long options are mandatory for short options too. Startup: -V, --version display the version of Wget and exit. -h, --help print this help. -b, --background go to background after startup. -e, --execute=COMMAND execute a `.wgetrc'-style command. Logging and input file: -o, --output-file=FILE log messages to FILE. -a, --append-output=FILE append messages to FILE. -d, --debug print lots of debugging information. -q, --quiet quiet (no output). -v, --verbose be verbose (this is the default). -nv, --no-verbose turn off verboseness, without being quiet. -i, --input-file=FILE download URLs found in FILE. -F, --force-html treat input file as HTML. -B, --base=URL prepends URL to relative links in -F -i file. Download: -t, --tries=NUMBER set number of retries to NUMBER (0 unlimits). --retry-connrefused retry even if connection is refused. -O, --output-document=FILE write documents to FILE. -nc, --no-clobber skip downloads that would download to existing files. -c, --continue resume getting a partially-downloaded file. --progress=TYPE select progress gauge type. -N, --timestamping don't re-retrieve files unless newer than local. -S, --server-response print server response. --spider don't download anything. -T, --timeout=SECONDS set all timeout values to SECONDS. --dns-timeout=SECS set the DNS lookup timeout to SECS. --connect-timeout=SECS set the connect timeout to SECS. --read-timeout=SECS set the read timeout to SECS. -w, --wait=SECONDS wait SECONDS between retrievals. --waitretry=SECONDS wait 1..SECONDS between retries of a retrieval. --random-wait wait from 0...2*WAIT secs between retrievals. -Y, --proxy explicitly turn on proxy. --no-proxy explicitly turn off proxy. -Q, --quota=NUMBER set retrieval quota to NUMBER. --bind-address=ADDRESS bind to ADDRESS (hostname or IP) on local host. --limit-rate=RATE limit download rate to RATE. --no-dns-cache disable caching DNS lookups. --restrict-file-names=OS restrict chars in file names to ones OS allows. --user=USER set both ftp and http user to USER. --password=PASS set both ftp and http password to PASS. Directories: -nd, --no-directories don't create directories. -x, --force-directories force creation of directories. -nH, --no-host-directories don't create host directories. --protocol-directories use protocol name in directories. -P, --directory-prefix=PREFIX save files to PREFIX/... --cut-dirs=NUMBER ignore NUMBER remote directory components. HTTP options: --http-user=USER set http user to USER. --http-password=PASS set http password to PASS. --no-cache disallow server-cached data. -E, --html-extension save HTML documents with `.html' extension. --ignore-length ignore `Content-Length' header field. --header=STRING insert STRING among the headers. --proxy-user=USER set USER as proxy username. --proxy-password=PASS set PASS as proxy password. --referer=URL include `Referer: URL' header in HTTP request. --save-headers save the HTTP headers to file. -U, --user-agent=AGENT identify as AGENT instead of Wget/VERSION. --no-http-keep-alive disable HTTP keep-alive (persistent connections). --no-cookies don't use cookies. --load-cookies=FILE load cookies from FILE before session. --save-cookies=FILE save cookies to FILE after session. --keep-session-cookies load and save session (non-permanent) cookies. --post-data=STRING use the POST method; send STRING as the data. --post-file=FILE use the POST method; send contents of FILE. HTTPS (SSL/TLS) options: --secure-protocol=PR choose secure protocol, one of auto, SSLv2, SSLv3, and TLSv1. --no-check-certificate don't validate the server's certificate. --certificate=FILE client certificate file. --certificate-type=TYPE client certificate type, PEM or DER. --private-key=FILE private key file. --private-key-type=TYPE private key type, PEM or DER. --ca-certificate=FILE file with the bundle of CA's. --ca-directory=DIR directory where hash list of CA's is stored. --random-file=FILE file with random data for seeding the SSL PRNG. --egd-file=FILE file naming the EGD socket with random data. FTP options: --ftp-user=USER set ftp user to USER. --ftp-password=PASS set ftp password to PASS. --no-remove-listing don't remove `.listing' files. --no-glob turn off FTP file name globbing. --no-passive-ftp disable the "passive" transfer mode. --retr-symlinks when recursing, get linked-to files (not dir). --preserve-permissions preserve remote file permissions. Recursive download: -r, --recursive specify recursive download. -l, --level=NUMBER maximum recursion depth (inf or 0 for infinite). --delete-after delete files locally after downloading them. -k, --convert-links make links in downloaded HTML point to local files. -K, --backup-converted before converting file X, back up as X.orig. -m, --mirror shortcut for -N -r -l inf --no-remove-listing. -p, --page-requisites get all images, etc. needed to display HTML page. --strict-comments turn on strict (SGML) handling of HTML comments. Recursive accept/reject: -A, --accept=LIST comma-separated list of accepted extensions. -R, --reject=LIST comma-separated list of rejected extensions. -D, --domains=LIST comma-separated list of accepted domains. --exclude-domains=LIST comma-separated list of rejected domains. --follow-ftp follow FTP links from HTML documents. --follow-tags=LIST comma-separated list of followed HTML tags. --ignore-tags=LIST comma-separated list of ignored HTML tags. -H, --span-hosts go to foreign hosts when recursive. -L, --relative follow relative links only. -I, --include-directories=LIST list of allowed directories. -X, --exclude-directories=LIST list of excluded directories. -np, --no-parent don't ascend to the parent directory. Mail bug reports and suggestions to <bug-wget@gnu.org>.

Is there going to be a sorting function next time?

　　Brother, what do you mean by sorting?

I'm going to add an automatic generation of the pac configuration file~~ haha

　　I don't understand the format of the pac file, please give me some guidance.

First, say sorry to brother vkill~ When I was clicking to see, I accidentally clicked there. Out of curiosity, I took a look, and as a result, I deducted one point from you. You can also deduct one point from me, or I'll add one point back to you when I see your post some day.

Sorting means that the user specifies a web page to access, and then displays available proxies according to the speed of connecting to this web page.

Great! Just tried it and it's wonderful! I've given you extra points, brother.

Floor1　lxmxn	Posted 2006-11-27 12:27
版主　Posts 4,938　Credits 11,386
In daily internet surfing, sometimes proxy servers are used. The usual way is to go to some websites that provide proxy server lists to find the proxies we want. But there are many proxies above that are unavailable.难道要 manually test one by one which proxy is available? Our batch script can complete this task. 　　Using the proxy function of wget, download a fixed web page, such as www.baidu.com. I downloaded the web page www.ip138.com to achieve the purpose of simply detecting whether the proxy server lists of some websites are available. The website I used to provide the proxy server list is: http://www.cnproxy.com/proxyedu1.html 　　This batch processing cannot perfectly achieve our purpose. There are still many parameters and formats that can be modified. Everyone can modify the statements and parameters in it according to their own wishes. Everyone can also post their opinions if they have any. This batch processing is best run in a folder because many files will be generated in the middle. 　　Regarding the small tool wget, you can search in the forum to find it. 　　Here, I would like to thank Brother electronixtar for the proposal and Brother redtek for the help. @echo off&setlocal ENABLEDELAYEDEXPANSION&title 代理检测 if exist proxyedu1.html del proxyedu1.html /a/s >nul if exist useproxy.txt del /a/q useproxy.txt echo 于 %date% %time% 检测 http://www.cnproxy.com/proxyedu1.html 可用代理如下: >useproxy.txt echo;>>useproxy.txt echo ############################################################# >>useproxy.txt wget http://www.cnproxy.com/proxyedu1.html >nul 2>nul for /f "tokens=1,2 delims=^<trd>: " %%a in ('type proxyedu1.html ^\| findstr /c:"HTTP" ') do ( set set= set set=%%a set "set=!set:&#8205=!" set "set=!set!:%%b" echo 正在检测 !set! 代理，请稍等.... wget -q -T 3 -t 1 -e "http_proxy=!set!" http://www.ip138.com -O - >nul 2>nul && echo 检测到 ***** !set! ***** 代理可用&&echo ----!set! >>useproxy.txt ) start useproxy.txt goto :eof [ Last edited by lxmxn on 2006-11-27 at 01:01 PM ]

Floor2　无奈何	Posted 2006-11-27 12:49
荣誉版主　Posts 356　Credits 1,338
Not bad, not bad. We should dig deeper into the functions of wget. Here are a few points to supplement about wget: wget -q does not output download record information wget -O - outputs the downloaded content to the screen instead of writing to a file

Floor3　lxmxn	Posted 2006-11-27 12:53
版主　Posts 4,938　Credits 11,386
Thanks for the moderator's guidance. 　　Since I just started learning to use wget, I haven't discovered so many useful parameters yet. Even the -t parameter was only found out after searching Baidu. Sweat. 　　I'll improve the code again.

Floor4　redtek	Posted 2006-11-27 22:06
金牌会员　Posts 1,147　Credits 2,902
Such a fun code, it's practical and provides more convenience for downloading~ :))) Appreciate it~~

Floor5　vkill	Posted 2006-11-27 22:38
金牌会员　Posts 1,744　Credits 4,103　From 甘肃.临泽
I also tried wget before && didn't succeed, maybe I didn't add parameters!

Floor6　electronixtar	Posted 2006-11-27 23:29
铂金会员　Posts 2,672　Credits 7,493
Top it's a boon for users of our education network!

Floor7　electronixtar	Posted 2006-11-27 23:52
铂金会员　Posts 2,672　Credits 7,493
Forgot to add points, original work is hard, highly recommend

Floor8　vkill	Posted 2006-11-28 00:28
金牌会员　Posts 1,744　Credits 4,103　From 甘肃.临泽
The automatic proxy setup done 2 months ago was a pity because there was no way to verify it. Now heh~ thanks

Floor9　NaturalJ0	Posted 2006-11-28 00:41
银牌会员　Posts 533　Credits 1,181
What a good thing. Will you be preparing to add a sorting function next time?

Floor10　electronixtar	Posted 2006-11-28 01:01
铂金会员　Posts 2,672　Credits 7,493
When joining an automatic generation of pac configuration file ~~ haha

Floor11　a9319751	Posted 2006-11-28 01:01
中级用户　Posts 170　Credits 439
GNU Wget 1.10.2, a non-interactive network retriever. Usage: wget ... ... Mandatory arguments to long options are mandatory for short options too. Startup: -V, --version display the version of Wget and exit. -h, --help print this help. -b, --background go to background after startup. -e, --execute=COMMAND execute a `.wgetrc'-style command. Logging and input file: -o, --output-file=FILE log messages to FILE. -a, --append-output=FILE append messages to FILE. -d, --debug print lots of debugging information. -q, --quiet quiet (no output). -v, --verbose be verbose (this is the default). -nv, --no-verbose turn off verboseness, without being quiet. -i, --input-file=FILE download URLs found in FILE. -F, --force-html treat input file as HTML. -B, --base=URL prepends URL to relative links in -F -i file. Download: -t, --tries=NUMBER set number of retries to NUMBER (0 unlimits). --retry-connrefused retry even if connection is refused. -O, --output-document=FILE write documents to FILE. -nc, --no-clobber skip downloads that would download to existing files. -c, --continue resume getting a partially-downloaded file. --progress=TYPE select progress gauge type. -N, --timestamping don't re-retrieve files unless newer than local. -S, --server-response print server response. --spider don't download anything. -T, --timeout=SECONDS set all timeout values to SECONDS. --dns-timeout=SECS set the DNS lookup timeout to SECS. --connect-timeout=SECS set the connect timeout to SECS. --read-timeout=SECS set the read timeout to SECS. -w, --wait=SECONDS wait SECONDS between retrievals. --waitretry=SECONDS wait 1..SECONDS between retries of a retrieval. --random-wait wait from 0...2*WAIT secs between retrievals. -Y, --proxy explicitly turn on proxy. --no-proxy explicitly turn off proxy. -Q, --quota=NUMBER set retrieval quota to NUMBER. --bind-address=ADDRESS bind to ADDRESS (hostname or IP) on local host. --limit-rate=RATE limit download rate to RATE. --no-dns-cache disable caching DNS lookups. --restrict-file-names=OS restrict chars in file names to ones OS allows. --user=USER set both ftp and http user to USER. --password=PASS set both ftp and http password to PASS. Directories: -nd, --no-directories don't create directories. -x, --force-directories force creation of directories. -nH, --no-host-directories don't create host directories. --protocol-directories use protocol name in directories. -P, --directory-prefix=PREFIX save files to PREFIX/... --cut-dirs=NUMBER ignore NUMBER remote directory components. HTTP options: --http-user=USER set http user to USER. --http-password=PASS set http password to PASS. --no-cache disallow server-cached data. -E, --html-extension save HTML documents with `.html' extension. --ignore-length ignore `Content-Length' header field. --header=STRING insert STRING among the headers. --proxy-user=USER set USER as proxy username. --proxy-password=PASS set PASS as proxy password. --referer=URL include `Referer: URL' header in HTTP request. --save-headers save the HTTP headers to file. -U, --user-agent=AGENT identify as AGENT instead of Wget/VERSION. --no-http-keep-alive disable HTTP keep-alive (persistent connections). --no-cookies don't use cookies. --load-cookies=FILE load cookies from FILE before session. --save-cookies=FILE save cookies to FILE after session. --keep-session-cookies load and save session (non-permanent) cookies. --post-data=STRING use the POST method; send STRING as the data. --post-file=FILE use the POST method; send contents of FILE. HTTPS (SSL/TLS) options: --secure-protocol=PR choose secure protocol, one of auto, SSLv2, SSLv3, and TLSv1. --no-check-certificate don't validate the server's certificate. --certificate=FILE client certificate file. --certificate-type=TYPE client certificate type, PEM or DER. --private-key=FILE private key file. --private-key-type=TYPE private key type, PEM or DER. --ca-certificate=FILE file with the bundle of CA's. --ca-directory=DIR directory where hash list of CA's is stored. --random-file=FILE file with random data for seeding the SSL PRNG. --egd-file=FILE file naming the EGD socket with random data. FTP options: --ftp-user=USER set ftp user to USER. --ftp-password=PASS set ftp password to PASS. --no-remove-listing don't remove `.listing' files. --no-glob turn off FTP file name globbing. --no-passive-ftp disable the "passive" transfer mode. --retr-symlinks when recursing, get linked-to files (not dir). --preserve-permissions preserve remote file permissions. Recursive download: -r, --recursive specify recursive download. -l, --level=NUMBER maximum recursion depth (inf or 0 for infinite). --delete-after delete files locally after downloading them. -k, --convert-links make links in downloaded HTML point to local files. -K, --backup-converted before converting file X, back up as X.orig. -m, --mirror shortcut for -N -r -l inf --no-remove-listing. -p, --page-requisites get all images, etc. needed to display HTML page. --strict-comments turn on strict (SGML) handling of HTML comments. Recursive accept/reject: -A, --accept=LIST comma-separated list of accepted extensions. -R, --reject=LIST comma-separated list of rejected extensions. -D, --domains=LIST comma-separated list of accepted domains. --exclude-domains=LIST comma-separated list of rejected domains. --follow-ftp follow FTP links from HTML documents. --follow-tags=LIST comma-separated list of followed HTML tags. --ignore-tags=LIST comma-separated list of ignored HTML tags. -H, --span-hosts go to foreign hosts when recursive. -L, --relative follow relative links only. -I, --include-directories=LIST list of allowed directories. -X, --exclude-directories=LIST list of excluded directories. -np, --no-parent don't ascend to the parent directory. Mail bug reports and suggestions to <bug-wget@gnu.org>.

Floor12　lxmxn	Posted 2006-11-28 03:44
版主　Posts 4,938　Credits 11,386
Is there going to be a sorting function next time? 　　Brother, what do you mean by sorting? I'm going to add an automatic generation of the pac configuration file~~ haha 　　I don't understand the format of the pac file, please give me some guidance.

Floor13　snowbbbb	Posted 2006-11-28 03:48
初级用户　Posts 42　Credits 118
First, say sorry to brother vkill~ When I was clicking to see, I accidentally clicked there. Out of curiosity, I took a look, and as a result, I deducted one point from you. You can also deduct one point from me, or I'll add one point back to you when I see your post some day.

Floor14　NaturalJ0	Posted 2006-11-28 04:32
银牌会员　Posts 533　Credits 1,181
Sorting means that the user specifies a web page to access, and then displays available proxies according to the speed of connecting to this web page.

Floor15　ccwan	Posted 2006-11-28 04:46
金牌会员　Posts 1,160　Credits 2,725　From 河北廊坊
Great! Just tried it and it's wonderful! I've given you extra points, brother.