Mass batch URL downloader needed...

[ld13]

I am looking for an alternative to Free Download Manager that allows more control over batched url downloads.

In the past I have used FDM for batched downloads by giving it the url http://server/docs/fileid_(*).pdf and assigning it the range 000-999. It would then download all the available files from fileid_000.pdf to fileid_999.pdf. The problem with this was that any increase in the range like 0000-9999 would take significantly longer to generate and the program would freeze up for 'hours' before it eventually started downloading the links. It would even crash if the range was ridiculously large.

I am now sitting with a server where the range of files to be extracted are indeed ridiculous and I do not have FTP access as the company thought they would save themselves R30 a month and get a free hosting provider. In the past I would have pre-generated the range of links in excel and then painstakingly copied a few hundred links at a time but this it quite time consuming.

Does anyone know of an alternative method/program that would, in a sense, allow me to batch a batched download in such a way that eg. only 1/100 links get generated and downloaded automatically at a time before the next batch is processed?

Thx

 

[bruce_the_loon]

DownThemAll addon for Firefox, I added a URL with 0000-9999 wildcard and it took 5 seconds to add them all. You can set the maximum concurrent downloads in total and per server as well as number of segments downloaded at the same time.

URL - https://addons.mozilla.org/en-US/firefox/addon/downthemall/

 

[froot]

This. Downthemall grabs all the links and then allows you to select what you want to grab (with checkboxes) and you can filter via different filetypes. It's an awesome program.

 

[Spudman]

Internet Download Manager.
Nice interface, cool batch downloader / site grabber.

 

[ld13]

You guys are awesome! Thx

I've had DownThemAll since forever but did not know it had this functionality built in. I just added 100 000 links within a minute while FF ate about 1.2GB of ram without hesitating. Had a few issues to get it started but after adjusting for duplicate files, setting the correct amount of allowed concurrent connections and adding the correct referrer everything seems to be working absolutely beautifully!

Now I just need to run through all 100 000 files to check that the contents matches the file extension. Everything is .txt but a few .html and .pdf files are supposed to be amongst them - busy building a command line .bat script should be able to find and rename the files based on their content... unless someone knows of any program that might be able to do such a thing?

 

[bullzeye.za]

Get yourself a bash terminal ( Cygwin works in windows ) and..

for i in `seq 100 9999`; do wget -c http://fileserver.com/$i.pdf; done

 

[ponder]

Get yourself a bash terminal ( Cygwin works in windows ) and..

for i in `seq 100 9999`; do wget -c http://fileserver.com/$i.pdf; done

Heathen! Expecting people to use linux black magic

 

[ld13]

Heathen! Expecting people to use linux black magic

There is nothing wrong with using bash et al. Although the last time I used it was with my eggdrop about 5+ years ago.

 

[ponder]

Although the last time I used it was with my eggdrop about 5+ years ago.

Time for you to be born again...