Batch Intelligent Download of Web Page Images - China DOS Union Forum

### Part 1
The first part of the code is:
```

curl "http://220.181.38.82/m?ct=134217728&tn=baidusg,不得不爱  &word=mp3,http://www.yantaiblog.net/UploadFiles/2007-7/aGhmbGtmbGsx.mp3,,&si=%B2%BB%B5%C3%B2%BB%B0%AE;;%C5%CB%E7%E2%B0%D8;;51913;;51913&lm=16777216"

curl: (3) illegal character in range specification at pos 132
总感觉curl在某些方面有很多缺陷，为何出现这样的情况？
有时 url中有 & 字符时必需使用在 & 前加转义字符，而有些时候却不需要，为何？

The translated text for the first part is:
The first code block is:
```

curl "http://220.181.38.82/m?ct=134217728&tn=baidusg,不得不爱  &word=mp3,http://www.yantaiblog.net/UploadFiles/2007-7/aGhmbGtmbGsx.mp3,,&si=%B2%BB%B5%C3%B2%BB%B0%AE;;%C5%CB%E7%E2%B0%D8;;51913;;51913&lm=16777216"

curl: (3) illegal character in range specification at pos 132
I always feel that curl has many defects in some aspects. Why does such a situation occur?
Sometimes when there are & characters in the URL, it is necessary to add an escape character before &, but sometimes it is not necessary. Why?

### Part 2
The second part of the code is:

curl http://list.mp3.baidu.com/topso/mp3topsong.html | sed "s/\x22/\xa/g;/baidump3/!d"

curl http://list.mp3.baidu.com/topso/mp3topsong.html | sed "s/\x22/\xa/g" | sed "/baidump3/!d"

这2个命令执行的结果为何不一样？

The translated text for the second part is:
The second code block is:

curl http://list.mp3.baidu.com/topso/mp3topsong.html | sed "s/\x22/\xa/g;/baidump3/!d"

curl http://list.mp3.baidu.com/topso/mp3topsong.html | sed "s/\x22/\xa/g" | sed "/baidump3/!d"

Why do the results of these two commands differ?

I always feel that curl has many drawbacks in some aspects. Why does this happen?
Sometimes when there is an & character in the url, an escape character must be added before &, but sometimes it is not necessary. Why?
--------------------------
1 I think it should be related to the CMD.EXE syntax analysis. When cmd encounters &, it considers it as the end of a statement.

Why are the results of these two commands different?
--------------------------------------------
2. sed reads data line by line. The general process is to extract a line of data from curl, read the content and store it in the pattern space. After processing with sed "s/\x22/\xa/g", it seems to generate multiple lines. In fact, it is still stored in the pattern space as an unprocessed semi-finished product, and then processed by /baidump3/!d.
Anyway, remember
sed processes its input data line by line. The input may be in the form of pipes, files, communication in processes, etc.
The content of the pattern space is not treated in the same way as the input.

[ Last edited by junchen2 on 2008-3-17 at 01:54 PM ]

Like the web page submitted by from can be solved with both wget and curl, just try both.

D:\>wget "http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/2007-7/aGhmbGtmbGsx.mp3"
--17:52:54-- http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/200
7-7/aGhmbGtmbGsx.mp3
=> `u@u=http:%2F%2Fwww.yantaiblog.net%2FUploadFiles%2F2007-7%2FaGhmbG
tmbGsx.mp3'
Resolving mp3.baidu.com... done.
Connecting to mp3.baidu.com:80... connected.
HTTP request sent, awaiting response... 302 Object moved
Location: http://www.yantaiblog.net/UploadFiles/2007-7/76386053.mp3
--17:52:55-- http://www.yantaiblog.net/UploadFiles/2007-7/76386053.mp3
=> `76386053.mp3'
Resolving www.yantaiblog.net... done.
Connecting to www.yantaiblog.net:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4,505,113

0% 19,433 9.56K/s ETA 07:38

Can the LZ explain under what circumstances the keepalive option of curl should be used and how to use it? Thanks

Originally posted by zerocq at 2008-3-18 19:12:
Can the landlord explain when and how to use the keepalive option of curl?

Thank you

I haven't used the keepalive method
Reference


curl --manual

http://curl.haxx.se/docs/manpage.html

--keepalive-time <seconds>

This option sets the time a connection needs to remain idle before sending keepalive probes and the time between individual keepalive probes. It is currently effective on operating systems offering the TCP_KEEPIDLE and TCP_KEEPINTVL socket options (meaning Linux, recent AIX, HP-UX and more). This option has no effect if --no-keepalive is used. (Added in 7.18.0)

If this option is used multiple times, the last occurrence sets the amount.

[ Last edited by ngd on 2008-4-19 at 05:15 PM ]

Originally posted by knoppix7 at 2008-3-18 19:02:
D:\>wget "http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/2007-7/
aGhmbGtmbGsx.mp3"
--17:52:54-- http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/2 ... Strange! wget 1.10.2 D:\>wget "http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/2007-7/ aGhmbGtmbGsx.mp3" --20:32:27-- http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/200 7-7/aGhmbGtmbGsx.mp3 => `u@u=http%3A%2F%2Fwww.yantaiblog.net%2FUploadFiles%2F2007-7%2FaGhm bGtmbGsx.mp3' Resolving mp3.baidu.com... 220.181.38.70 Connecting to mp3.baidu.com|220.181.38.70|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://www.baidu.com/search/error.html --20:32:28-- http://www.baidu.com/search/error.html => `error.html' Resolving www.baidu.com... 220.181.6.6, 220.181.37.55 Connecting to www.baidu.com|220.181.6.6|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 2,699 (2.6K) 100% 2,699 --.--K/s 20:32:38 (52.95 MB/s) - `error.html' saved

D:\>wget --help
GNU Wget 1.8.2, a non-interactive network retriever.

curl -d "UserName=***&Password=******" http://www.netyi.net/jsLoginStatus.aspx Why can't I log in?

Floor16　ngd	Posted 2008-03-17 00:48
中级用户　Posts 108　Credits 312
### Part 1 The first part of the code is: ``` curl "http://220.181.38.82/m?ct=134217728&tn=baidusg,不得不爱 &word=mp3,http://www.yantaiblog.net/UploadFiles/2007-7/aGhmbGtmbGsx.mp3,,&si=%B2%BB%B5%C3%B2%BB%B0%AE;;%C5%CB%E7%E2%B0%D8;;51913;;51913&lm=16777216" curl: (3) illegal character in range specification at pos 132 总感觉curl在某些方面有很多缺陷，为何出现这样的情况？有时 url中有 & 字符时必需使用在 & 前加转义字符，而有些时候却不需要，为何？ The translated text for the first part is: The first code block is: ``` curl "http://220.181.38.82/m?ct=134217728&tn=baidusg,不得不爱 &word=mp3,http://www.yantaiblog.net/UploadFiles/2007-7/aGhmbGtmbGsx.mp3,,&si=%B2%BB%B5%C3%B2%BB%B0%AE;;%C5%CB%E7%E2%B0%D8;;51913;;51913&lm=16777216" curl: (3) illegal character in range specification at pos 132 I always feel that curl has many defects in some aspects. Why does such a situation occur? Sometimes when there are & characters in the URL, it is necessary to add an escape character before &, but sometimes it is not necessary. Why? ### Part 2 The second part of the code is: curl http://list.mp3.baidu.com/topso/mp3topsong.html \| sed "s/\x22/\xa/g;/baidump3/!d" curl http://list.mp3.baidu.com/topso/mp3topsong.html \| sed "s/\x22/\xa/g" \| sed "/baidump3/!d" 这2个命令执行的结果为何不一样？ The translated text for the second part is: The second code block is: curl http://list.mp3.baidu.com/topso/mp3topsong.html \| sed "s/\x22/\xa/g;/baidump3/!d" curl http://list.mp3.baidu.com/topso/mp3topsong.html \| sed "s/\x22/\xa/g" \| sed "/baidump3/!d" Why do the results of these two commands differ?

Floor17　junchen2	Posted 2008-03-17 13:53
高级用户　Posts 219　Credits 537　From 杭州--半山
I always feel that curl has many drawbacks in some aspects. Why does this happen? Sometimes when there is an & character in the url, an escape character must be added before &, but sometimes it is not necessary. Why? -------------------------- 1 I think it should be related to the CMD.EXE syntax analysis. When cmd encounters &, it considers it as the end of a statement. Why are the results of these two commands different? -------------------------------------------- 2. sed reads data line by line. The general process is to extract a line of data from curl, read the content and store it in the pattern space. After processing with sed "s/\x22/\xa/g", it seems to generate multiple lines. In fact, it is still stored in the pattern space as an unprocessed semi-finished product, and then processed by /baidump3/!d. Anyway, remember sed processes its input data line by line. The input may be in the form of pipes, files, communication in processes, etc. The content of the pattern space is not treated in the same way as the input. [ Last edited by junchen2 on 2008-3-17 at 01:54 PM ]

Floor18　vkill	Posted 2008-03-18 16:24
金牌会员　Posts 1,744　Credits 4,103　From 甘肃.临泽
Like the web page submitted by from can be solved with both wget and curl, just try both.

Floor19　knoppix7	Posted 2008-03-18 19:02
银牌会员　Posts 634　Credits 1,287　From cmd.exe
D:\>wget "http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/2007-7/aGhmbGtmbGsx.mp3" --17:52:54-- http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/200 7-7/aGhmbGtmbGsx.mp3 => `u@u=http:%2F%2Fwww.yantaiblog.net%2FUploadFiles%2F2007-7%2FaGhmbG tmbGsx.mp3' Resolving mp3.baidu.com... done. Connecting to mp3.baidu.com:80... connected. HTTP request sent, awaiting response... 302 Object moved Location: http://www.yantaiblog.net/UploadFiles/2007-7/76386053.mp3 --17:52:55-- http://www.yantaiblog.net/UploadFiles/2007-7/76386053.mp3 => `76386053.mp3' Resolving www.yantaiblog.net... done. Connecting to www.yantaiblog.net:80... connected. HTTP request sent, awaiting response... 200 OK Length: 4,505,113 0% 19,433 9.56K/s ETA 07:38

Floor20　zerocq	Posted 2008-03-18 19:12
中级用户　Posts 196　Credits 458
Can the LZ explain under what circumstances the keepalive option of curl should be used and how to use it? Thanks

Floor21　ngd	Posted 2008-03-18 21:07
中级用户　Posts 108　Credits 312
Originally posted by zerocq at 2008-3-18 19:12: Can the landlord explain when and how to use the keepalive option of curl? Thank you I haven't used the keepalive method Reference curl --manual http://curl.haxx.se/docs/manpage.html --keepalive-time <seconds> This option sets the time a connection needs to remain idle before sending keepalive probes and the time between individual keepalive probes. It is currently effective on operating systems offering the TCP_KEEPIDLE and TCP_KEEPINTVL socket options (meaning Linux, recent AIX, HP-UX and more). This option has no effect if --no-keepalive is used. (Added in 7.18.0) If this option is used multiple times, the last occurrence sets the amount. [ Last edited by ngd on 2008-4-19 at 05:15 PM ]

Floor22　ngd	Posted 2008-03-18 21:46
中级用户　Posts 108　Credits 312
Originally posted by knoppix7 at 2008-3-18 19:02: D:\>wget "http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/2007-7/ aGhmbGtmbGsx.mp3" --17:52:54-- http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/2 ... Strange! wget 1.10.2 D:\>wget "http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/2007-7/ aGhmbGtmbGsx.mp3" --20:32:27-- http://mp3.baidu.com/u?u=http://www.yantaiblog.net/UploadFiles/200 7-7/aGhmbGtmbGsx.mp3 => `u@u=http%3A%2F%2Fwww.yantaiblog.net%2FUploadFiles%2F2007-7%2FaGhm bGtmbGsx.mp3' Resolving mp3.baidu.com... 220.181.38.70 Connecting to mp3.baidu.com\|220.181.38.70\|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://www.baidu.com/search/error.html --20:32:28-- http://www.baidu.com/search/error.html => `error.html' Resolving www.baidu.com... 220.181.6.6, 220.181.37.55 Connecting to www.baidu.com\|220.181.6.6\|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 2,699 (2.6K) 100% 2,699 --.--K/s 20:32:38 (52.95 MB/s) - `error.html' saved

Floor23　knoppix7	Posted 2008-03-19 19:41
银牌会员　Posts 634　Credits 1,287　From cmd.exe
D:\>wget --help GNU Wget 1.8.2, a non-interactive network retriever.

Floor24　kioskboy	Posted 2008-08-08 11:38
初级用户　Posts 103　Credits 153
curl -d "UserName=*&Password=****" http://www.netyi.net/jsLoginStatus.aspx Why can't I log in?