China DOS Union

How to download all web pages on this forum and use them offline?

[ Last edited by 740011611 on 2010-3-19 at 18:33 ]

I'm afraid it's not easy to achieve

Baidu search, full site download...

Thanks everyone, it's not easy to achieve! We can only do it manually.

I don't know if Wget can achieve the goal of downloading the entire site.
wget -r -p -np -E -k http://www.cn-dos.net/forum/

I found that many web pages are downloaded repeatedly. What should I do?

Help me? Moderator...

www.google.cn

Note:

It is not that we hope the original poster can immediately master the usage of wget or curl through Google search, but that we hope the original poster can search for tools like "whole site download" that have been seen everywhere for many years.

[ Last edited by HAT on 2010-3-26 at 17:17 ]

I've seen someone make a script to download all articles of a blog. It's done by downloading the main page, then analyzing the links in it (like links to the next article, next page, etc., extracting related links), and then downloading further, in a cyclical manner.

[ Last edited by 523066680 on 2010-3-16 at 13:44 ]

Moderator, can I search the entire site to download software? I don't know which one is easy to use? I just want to download the web pages of this forum. Moderator, give me a method? Urgent!

[ Last edited by 740011611 on 2010-3-19 at 18:34 ]

There is a green full-site download tool in my cloud disk (the one in my signature), which is the pro.rar in the C:\soft directory. It is relatively easy to use (not command-line). It comes with a keygen. After registration, you can download without restrictions.

Click on the file, the New Project Wizard, and then select the second option - Copy a website, including the directory structure of the website, and you can mirror a website.

=====================================================
Let me interject, you can all fire at me:

There are many software for downloading full sites online, but for a novice to use it well, it takes a process. Everyone can use search, but to master a professional software, one must understand some professional knowledge. Those wge, curl online should all be relatively powerful tools. I ask, how many experts in the forum say they have mastered them well? If you just tell a novice to use Google at every turn, it seems this problem is too simple. I ask, is our forum just for people to learn how to search?

The union has relatively few discussion threads about full-site download or wget, curl usage, and the only existing threads also stay at a very shallow depth.

======================================================

Originally posted by 740011611 at 2010-3-14 07:37 PM:
I don't know if Wget can achieve the goal of downloading the entire site?
wget -r -p -np -E -k http://www.cn-dos.net/forum/

I found that many web pages are downloaded repeatedly, what should I do?

When I use wget to download the entire site, I use a parameter

wget -m www.cn-dos.net/forum

You can give it a try

[ Last edited by plp626 on 2010-3-17 at 09:22 ]

I really appreciate your selfless help and tireless explanations! They have created a very good learning environment for us novices. Wish the DOS Union better and better!

Back to the topic: I used wget -m http://www.cn-dos/forum to download mostly duplicate files (note that they are not in web page format! It's not easy to take a screenshot here, so I didn't upload it to you!)
Finally, I still had to use for /f "delims=" %i in ('dir /a /b') do ren "%i" "%i.html" to rename.
But there are still many useless web pages. Do I still have to delete them one by one? Is there a better way?

Saw a post like this:
@echo off

::Download the essence index post
cd.>essence.txt
wget -q -O essence.txt "http://www.cn-dos.net/forum/viewthread.php?tid=27667"

::Set the file save directory here
set downdir=cn-dos forum essence posts
md %downdir%
echo\is downloading... Please wait...

::Batch download posts to the newly created directory
for /f %%a in ('sed -n "s/^\*\)}.*$/\1/p" essence.txt') do (
wget -q -O %downdir%\%%a.html "http://www.cn-dos.net/forum/viewthread.php?tid=%%a&action=printable"
)

::Download completed, open the directory
echo Download completed
ping -n 3 127.0.0.1 >nul&&start %downdir%



This download is very good. I don't know if it can be used here? Please trouble the experts to take a look.

You search for "Network Sniffer", but unfortunately the next post is a bit slow, all in HTML format.

If you pay attention usually, you will find that the addresses of the posts are all like this (the following few parameters are not important)

http://www.cn-dos.net/forum/viewthread.php?tid=Post number

Just enter it directly to enter a post, and fortunately, this number seems to increase in the way of natural numbers. Now it is almost 50574. If you add a parameter &action=printable, then a printable version will come out, that is, a page containing only the basic post content, which is the one you mentioned in building 13. You can probably choose a number to start and cycle to download.

But this method is extremely not recommended because the post name cannot be obtained, it is not known which section it is in, and it is not known which numbers are not available (some posts have been deleted)