suggestion for a Novice friendly web scraper

S

SpartanX

Well-Known Member
Joined
Jun 2, 2014
Messages
158
Reaction score
0
Hey Guys

I'm looking for an easy to use web scraper. I want something simple, not feature rich. What I want is, I'd like to be able to scrape products, prices, URL, images, etc of products from websites like kalahari.com. And be able to to export the file as CSV so I can edit it if i have to.

I'm a total beginner so I need something that i will insert a URL, press a few buttons and get results.

Free or paid I'd like to look at it. If paid, I hope it has a free trial.

Thanks for your help!
 
SpartanX said:
Hey Guys

I'm looking for an easy to use web scraper. I want something simple, not feature rich. What I want is, I'd like to be able to scrape products, prices, URL, images, etc of products from websites like kalahari.com. And be able to to export the file as CSV so I can edit it if i have to.

I'm a total beginner so I need something that i will insert a URL, press a few buttons and get results.

Free or paid I'd like to look at it. If paid, I hope it has a free trial.

Thanks for your help!
Click to expand...

Hey

Firstly, you will not find a one-for-all scraper, whether this is done from a webserver or via software.

Secondly, you will need to make a list of all websites you need scraped in order to provide this to a developer. The main reason being that different websites have different methods of delivering content, so the coding needs to be done for each website individually.

Lastly, I doubt that any developer will do this free of charge. There is a lot of coding involved just in the scraping alone.

Also, yes, there may be legal implications. As far as I know almost all these websites prevents scraping of their data in their TOS.

So, I suggest that you be more comprehensive in terms of how many websites and which ones you need scraped. This will give a coder an idea as to how much to quote you and how long this will take.
 
House said:
Hey

Firstly, you will not find a one-for-all scraper, whether this is done from a webserver or via software.

Secondly, you will need to make a list of all websites you need scraped in order to provide this to a developer. The main reason being that different websites have different methods of delivering content, so the coding needs to be done for each website individually.

Lastly, I doubt that any developer will do this free of charge. There is a lot of coding involved just in the scraping alone.

Also, yes, there may be legal implications. As far as I know almost all these websites prevents scraping of their data in their TOS.

So, I suggest that you be more comprehensive in terms of how many websites and which ones you need scraped. This will give a coder an idea as to how much to quote you and how long this will take.
Click to expand...

Thanks for your response House. But isnt there something like I described that is already available on the market?
 
Most sites (like ours) have APIs and data dumps available.

Scrapping of websites will almost always be against TOS and can land you in hot water especially if you use it for commercial purposes.

Most sites will eventually block your ip based on heuristics.


Sent from my iPhone using MyBroadband Tapatalk
 
As the others have mentioned above, this is a grey area. Firstly as MagicDude noted, check if the site has an API available which can provide the data you require. This data will usually be served up as XML which you can parse on your end. Whether the site will give you access to it (or if they even have one) will differ from site to site.

If your scrape isn't going to be resource intensive I doubt many webmasters will have an issue with it but from what you're describing I don't think you should use any other method except for a site-enabled API.

I am using a custom scraper which I put together using PHP and the simplehtmldom library, however I'm using it for one page and only running it once a day. It's also not exactly noob friendly and you'll require some basic knowledge of php and regex statements.
 
Last edited:
SpartanX said:
Thanks for your response House. But isnt there something like I described that is already available on the market?
Click to expand...

Hey Spartan

There are software available that will crawl and index pages, but will not deliver the results in the way you require it. Software will almost certainly index pages and save them as HTML pages and you will need to compile your own CSV file from thousands of pages.

What you have described, you will need custom software that will be able to index and record very specific information from each page.

Like others mentioned above, such script will be resource intensive and you are likely to have your IP banned on these sites, leaving you unable to scrape information any further.

That being said, a script like the one you require is possible - even one that will not ban your IP:

If you do get a good developer a web-based script can be written that will index a certain number of pages in a certain amount of time. The script can, for instance, crawl 5 pages in 3 second intervals, stop for 10 minutes and then index the same amount again, until all pages have been indexed. You can also code IP changes into the script that will change the script's IP address every "X" amount of crawls.

The results can then be stored in an online database and then allow you to print or download it in virtually any format, CSV, XLS, XLSX, Doc, etc.

However, these scripts do not come cheap and may take some time to complete. The biggest problem you may face is the website you are crawling making changes to URLs, fields or database tables. The script will need to be updated regularly to adapt to changes.

H
 
House said:
However, these scripts do not come cheap and may take some time to complete. The biggest problem you may face is the website you are crawling making changes to URLs, fields or database tables. The script will need to be updated regularly to adapt to changes.
Click to expand...

+1 to this.
 
House said:
Hey Spartan

There are software available that will crawl and index pages, but will not deliver the results in the way you require it. Software will almost certainly index pages and save them as HTML pages and you will need to compile your own CSV file from thousands of pages.

What you have described, you will need custom software that will be able to index and record very specific information from each page.

Like others mentioned above, such script will be resource intensive and you are likely to have your IP banned on these sites, leaving you unable to scrape information any further.

That being said, a script like the one you require is possible - even one that will not ban your IP:

If you do get a good developer a web-based script can be written that will index a certain number of pages in a certain amount of time. The script can, for instance, crawl 5 pages in 3 second intervals, stop for 10 minutes and then index the same amount again, until all pages have been indexed. You can also code IP changes into the script that will change the script's IP address every "X" amount of crawls.

The results can then be stored in an online database and then allow you to print or download it in virtually any format, CSV, XLS, XLSX, Doc, etc.

However, these scripts do not come cheap and may take some time to complete. The biggest problem you may face is the website you are crawling making changes to URLs, fields or database tables. The script will need to be updated regularly to adapt to changes.

H
Click to expand...

Hi House

Really appreciate you taking out your time to answer my questions. Thanks, I think I learned a lot more than I wanted to know. This seems like it's going to be a mission! lol
 
If you PM me with what you actually want to achieve with this, I can organise a bidorbuy feed for you. A couple of members on MyBB use those data dumps for various projects.
 
MagicDude4Eva said:
If you PM me with what you actually want to achieve with this, I can organise a bidorbuy feed for you. A couple of members on MyBB use those data dumps for various projects.
Click to expand...

Now that is very nice! It will save you a lot of time and headaches if you have a valid request for information from BOB.
 
SpartanX said:
Hi House

Really appreciate you taking out your time to answer my questions. Thanks, I think I learned a lot more than I wanted to know. This seems like it's going to be a mission! lol
Click to expand...

Hey SpartanX

No problem at all. Glad I could help.
 
MagicDude4Eva said:
If you PM me with what you actually want to achieve with this, I can organise a bidorbuy feed for you. A couple of members on MyBB use those data dumps for various projects.
Click to expand...

Any chance of a better trade feed generator :P

Perhaps some cookies to recommend ;)
 
Last edited:
There is software available for this but it's business software so it's very expensive. It's not "grey" at all and commonly used to check the opposition prices and offerings. Our company sells this. The software is called Kapow. Have a look at Kapowsoftware.com. Very easy to configure. You can configure macros for input (multiple sites and sources) and output and then automate the process.
 
Last edited:
Your programming language of choice will have a way of calling web sites and retrieving the pages. There is nothing dodgy about it. The dodgy part is accessing areas restricted and you illegally obtain the access credentials. Generally, if you retrieve the pages with a browser you should have no issues. It's exactly what browser software does. It's then up to you to make sense of the result(the first hard part) and deal with the train smash when they change the site structure(the second hard part)

The hard parts mentioned above are the reason for things called "web services" :)
 
koeksGHT said:
Any chance of a better trade feed generator :P

Perhaps some cookies to recommend ;)
Click to expand...

We have something in the pipeline, but have not gotten around it yet (always seems to drop to the bottom of the list). Have not seen your PM BTW (mail me direct cto [at] bidorbuy[dot]co[dot]za)
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X