China DOS Union

Right now there are more than 300,000 htm files (and a small number of html files). A considerable portion of these files have been defaced with malware. It is found that the source files of these defaced html and htm files have the URL mm.aa88567.cn in the last line.

Now, we need to find those defaced web pages (whose last line contains the string "mm.aa88567.cn"), and then delete their last line.

Tools to be used: commonly used tools in Windows, or external tools: perl, sed,... (those with a volume size not exceeding 1M are all acceptable).

Now we need to find the tool with the fastest processing speed and its solution (code)

sed (40K version 4.0.7) download: http://upload.cn-dos.net/img/1813.rar
perl (369K version 5.005_03) download: http://upload.cn-dos.net/img/1814.rar

The following is the bat code for generating test files:
@echo off&setlocal enabledelayedexpansion md test&cd test echo Generating test files... for /l %%a in (1 1 2000)do ( (echo ^<html^> echo ...!random!... echo ^<... http://mm.aa88567.cn/ ... ^> )>!random!!time:~-2!!random!.htm ) for /l %%a in (1 1 3000)do ( (echo ^<html^> echo ...!random!... echo dfddfd )>!random!!time:~-2!!random!!time:~-2!.html ) set ff=!random! md %ff% for /l %%a in (1 1 2000)do ( (echo ^<html^> echo ...!random!... echo ^<... http://mm.aa88567.cn/ ... ^> )>%ff%\!random!!time:~-2!!random!.htm ) for /l %%a in (1 1 3000)do ( (echo ^<html^> echo ...!random!... echo dfddfd )>%ff%\!random!!time:~-2!!random!!time:~-2!.html )

[ Last edited by plp626 on 2010-3-4 at 23:00 ]

Do you want the WIN32 one?
Speed is unknown

Try
for /f "delims=" %%i in ('dir /s /b *.html') do sed -i "${/mm\.aa88567\.cn/d}" "%%i"

[ Last edited by Pierre on 2010-3-3 at 02:23 ]

```
sed -i "/mm\.aa88567\.cn/d" *.html
```

Just took a look, it's not simple. Can only help to give a thumbs up.

The code on floor 4 should be relatively fast ---- compared to the 3-way, there are no frequent starts of sed

However:

Generate a test file using the bat code on floor 1 in the test directory. Open cmd under the parent directory of test, and type sed -i "/mm\.aa88567\.cn/d" test\*.html
After that, in less than a few seconds, an error that sed cannot read memory appears

It seems that sed supports batch processing of at most 512 files, because when 511 temporary files are generated in the directory, this error dialog pops up

I don't know why, continue to pay attention....

[ Last edited by plp626 on 2010-3-3 at 15:11 ]

Directly using sed is the fastest. Don't use for loops, as for loops are the least efficient. Frequent opening and closing of sed processes also consumes a great deal of system resources. It's estimated that the CPU/memory will run out soon. Also, processing 300,000 files one by one won't be too fast, after all, it involves disk operations, unlike computational tasks done in memory.

Originally posted by plp626 at 2010-3-3 15:09:
The code on floor 4 should be faster---- compared to floor 3, there is no frequent startup of sed
But:

Use the bat code on floor 1 to generate test files in the test directory, open cmd under the parent directory path of test, and type sed -i &q ...

This still depends on the sed documentation. If there is a limit on the number of files edited simultaneously, then you can first divide all files into several directories by quantity through batch, and then perform sed processing on each directory in turn.

Nowadays, memory is very large. Install a ramdisk, move files to the ramdisk, and then use batch processing. The access speed of small files is extremely amazing. It is almost the same as executing in memory.

Directly using *.html, the files matched by the wildcard are all passed to sed as parameters for execution, so it cannot exceed a certain amount.
——If you use sed like this under the shell, the following will appear
-bash: /bin/sed: Argument list too long
Such an error.
Of course, it's not just for sed, but also for awk/grep, etc.
If you want higher efficiency, you can split the list and send 500 at a time to sed for processing

[ Last edited by Pierre on 2010-3-3 at 17:20 ]

Alternatively, directly use Perl to handle it and let it loop through the processing. Unfortunately, I'm not very familiar with using it. I'll try to write a solution later...

I don't know Perl very well either, and I really hope some expert can try using the Perl provided in the first floor.
Looking forward...

Post a self-used JS file replacement code, which has been modified according to the building owner's requirements.

if(WScript.arguments.length==0)PathSpec=get_path(); else PathSpec=WScript.arguments(0); fso=new ActiveXObject("Scripting.FileSystemObject"); WshShell=WScript.CreateObject("WScript.Shell"); WshShell.CurrentDirectory=fso.GetParentFolderName(WScript.ScriptFullName); if(!fso.FolderExists("#ReplacedFiles#"))fso.CreateFolder("#ReplacedFiles#"); WshShell.CurrentDirectory="#ReplacedFiles#" Main(PathSpec); WScript.quit(); function Main(FileSpec){ var fld,fs,fds,f,fd,curdir; curdir=fso.GetBaseName(FileSpec); if(!fso.FolderExists(curdir))fso.CreateFolder(curdir); curdir=fso.GetAbsolutePathName(curdir); WshShell.CurrentDirectory=curdir; fld = fso.getfolder(FileSpec); fds = new Enumerator(fld.subfolders); fs = new Enumerator(fld.files) for(;!fs.atEnd();fs.moveNext()){ f=fs.item();if(f.size==0)continue; if(/^html?$/gi.test(fso.getextensionname(f.name).toLowerCase())){ try{ var fl=fso.opentextfile(f.path,1); var sss=fl.readall(); fl.close(); }catch(err){ WScript.quit(); } var fl=fso.opentextfile(fso.GetBaseName(f.path)+".txt",2,true); sss=sss.replace(/.*mm\.aa88567\.cn.*\s*$/,""); fl.write(sss); fl.close(); } } for(;!fds.atEnd();fds.moveNext()){ d=fds.item(); Main(d.path) WshShell.CurrentDirectory=curdir } } function get_path(){ var objShell = new ActiveXObject("Shell.Application") do{ var objFolder = objShell.BrowseForFolder(0, "请选择文件夹:",0x301,0x11) if(objFolder == null)WScript.quit() var objPath = objFolder.Self.Path; if(/^:\\.+$/gi.test(objPath))break; }while(true) return objPath; }

Another AHK script

FileSelectFolder, sPath ,,3,Please select the folder you want to process: if not %errorlevel% { IfNotExist,%sPath%_BAK { MsgBox,4,,No backup found, do you want to create a backup? IfMsgBox Yes { TrayTip,,Creating backup...,5 FileCopyDir %sPath% , %sPath%_BAK } } Loop, %sPath%\*.html, , 1 ; Recurse into subfolders. { FileRead, sText, %A_LoopFileFullPath% sss := RegExReplace( sText ,".*mm\.aa88567\.cn.*\s$", "") FileDelete %A_LoopFileFullPath% FileAppend %sss%, %A_LoopFileFullPath% } } loop 2 { soundBeep 2500,500 sleep 500 }

I tried the code on floor 13. The speed was really good at that time, but the file extensions were all uniformly changed to txt. Those web files, the indexed ones have the suffix html and the non-indexed ones have htm.

I don't understand JS. How to modify the code so that the file extensions remain unchanged?