suggestion for a Novice friendly web scraper

everything to do with scraping is custom. there's no cookie cutter method because no 1 web page has the exact same predictable HTML code backing it. As mentioned, once their layout changes (even slightly) or they make a change to their url structure, you have to revisit your code again to make the change to have the scraper recognize it properly again
 
You could do this with our product - you'll need to string together a bunch of elements in a workflow that read a page, use regex to enumerate links and then read sub-pages.

Some of the guys at the office put together scraping workflows for PnP and woolworths about a year back. We structured the data in a database and then did fuzzy-matched price comparisons.

You can build this free at www.flowgear.net/developers - if you sign up and send me your email address, we'll copy in the PnP and Woolworths stuff into your site so you can use it as a baseline.
 
daniel_jc said:
You could do this with our product - you'll need to string together a bunch of elements in a workflow that read a page, use regex to enumerate links and then read sub-pages.

Some of the guys at the office put together scraping workflows for PnP and woolworths about a year back. We structured the data in a database and then did fuzzy-matched price comparisons.

You can build this free at www.flowgear.net/developers - if you sign up and send me your email address, we'll copy in the PnP and Woolworths stuff into your site so you can use it as a baseline.
Click to expand...

I do hope that you guys have asked Woolies / PnP or anyone else you are scrapping for permission:

Woolies Terms and Conditions said:
No other use is permitted without our prior written consent. The unauthorised use, copying, reproduction, variation, modification or distribution of the content of this website, the uploading of any unlawful or damaging information or viral software or the creation of any links to our website from any other site whatsoever is strictly prohibited.
Click to expand...
 
daniel_jc said:
Does Google ask their permission?

https://www.google.co.za/?q=site:www.woolworths.co.za/store
Click to expand...

Yes - http://www.robotstxt.org/ (not that Woolworths has one, but PnP and other sites do). You will find that many companies block BOTs (not just at robots.txt level, but also at firewall level) where waste of traffic is not desired. Google, Bing and others are tolerated as it benefits the website.

Having said that, I doubt that any online website would not give you permission (or provide you with a data-feed) if there is mutual benefit (i.e. affiliate programme, referral traffic etc). In most cases your referral traffic will earn you money as well.

Last part to think about: If you scrape a site, the site owner or the product owner of the content can raise a complaint with both ISPA and DMCA for takedown notices, which become very painful if you are trying to monetize on a hobby project. Just because the scrapping is tolerated, does not mean it is allowed/legal.
 
It's basically the pricecheck model right. Scraping will be a part of data aggregation for the foreseeable future because you won't always be able to mine from structured data (eg. via API or feed).

Interesting anecodote - I read in The Google Story (http://www.amazon.com/The-Google-St...d=1403268701&sr=8-2&keywords=the+google+story) that in the early days they got calls from irate website owners questioning why Google was eating up their bandwidth. We've come a long way...
 
Reviving this thread for my similar requirement..

I want to custom scrape a few websites on-demand, nothing dodge just want to extract additional metadata for my.. uhm .. special linux distributions that's not getting scraped by any of the known Kodi scrapers.

Don't mind the effort & updates for when websites change as a big part of this is for me learning/understanding coding for web, same argument goes for a custom program vs some website that does the heavy lifting for me. Don't think the sites have web api's too, so will be scraping directly off index.html.

So, I code for a living (sql/db developer) and I know java well enough, but it seems a bit low-level for this. Maybe I'm just lazy but asking never hurt, the thread suggested php, but what language/framework would you suggest for easy/quick development of such a requirement?
 
Last edited:
Ctrl+u, ctrl+a, ctrl+c and then ctrl+v on your site. I think that's best.
 
evilstebunny said:
Reviving this thread for my similar requirement..

I want to custom scrape a few websites on-demand, nothing dodge just want to extract additional metadata for my.. uhm .. special linux distributions that's not getting scraped by any of the known Kodi scrapers.

Don't mind the effort & updates for when websites change as a big part of this is for me learning/understanding coding for web, same argument goes for a custom program vs some website that does the heavy lifting for me. Don't think the sites have web api's too, so will be scraping directly off index.html.

So, I code for a living (sql/db developer) and I know java well enough, but it seems a bit low-level for this. Maybe I'm just lazy but asking never hurt, the thread suggested php, but what language/framework would you suggest for easy/quick development of such a requirement?
Click to expand...

Here is some quick and dirty php that's been running smoothly for me for a long time, I've just removed some of the site info:

Code: 
<?php
include('/var/www/admin/simplehtmldom/simple_html_dom.php');

// set timezone so we can convert later
date_default_timezone_set('Africa/Johannesburg');

// get DOM from URL or file
$html = file_get_html('http://www.sitexyz.com/Site.aspx');
// get raw contents for regex
$scrape = file_get_contents('http://www.sitexyz.com/Site.aspx');

// clean whitespace from scrape
$str = preg_replace('/\s+/', '', $scrape);

// match 7 day average
$regex_sd = '/7DayAverage:<\/td><tdwidth="50%">([0-9]+)<br\/>/';
if (preg_match($regex_sd, $str, $match_sd)) {
    $sd_value = $match_sd[1];
} else $sd_value = 0;

// find table with peak history
foreach($html->find('table[id=ctl00_ContentPlaceHolder2_GridView1]') as $e) {
    // echo $e->innertext;
}

$table = array();
$counter = 0;

foreach ($e->find('tr') as $row) {
    $date = $row->find('td',0)->plaintext;
    $time = $row->find('td',1)->plaintext;
    $peak = $row->find('td',2)->plaintext;

    $table[$counter]['date'] = $date;
    $table[$counter]['time'] = $time;
    $table[$counter]['peak'] = $peak;
    
    $counter++;
}

// echo values for debugging purposes

//print_r($table);
//echo $sd_value . "<br>";
//echo $now_value . "<br>";

// Connects to your Database 
// Some database stuff
?>
 
DrJohnZoidberg said:
Here is some quick and dirty php that's been running smoothly for me for a long time, I've just removed some of the site info:

Code: 
<?php
include('/var/www/admin/simplehtmldom/simple_html_dom.php');

// set timezone so we can convert later
date_default_timezone_set('Africa/Johannesburg');

// get DOM from URL or file
$html = file_get_html('http://www.sitexyz.com/Site.aspx');
// get raw contents for regex
$scrape = file_get_contents('http://www.sitexyz.com/Site.aspx');

// clean whitespace from scrape
$str = preg_replace('/\s+/', '', $scrape);

// match 7 day average
$regex_sd = '/7DayAverage:<\/td><tdwidth="50%">([0-9]+)<br\/>/';
if (preg_match($regex_sd, $str, $match_sd)) {
    $sd_value = $match_sd[1];
} else $sd_value = 0;

// find table with peak history
foreach($html->find('table[id=ctl00_ContentPlaceHolder2_GridView1]') as $e) {
    // echo $e->innertext;
}

$table = array();
$counter = 0;

foreach ($e->find('tr') as $row) {
    $date = $row->find('td',0)->plaintext;
    $time = $row->find('td',1)->plaintext;
    $peak = $row->find('td',2)->plaintext;

    $table[$counter]['date'] = $date;
    $table[$counter]['time'] = $time;
    $table[$counter]['peak'] = $peak;
    
    $counter++;
}

// echo values for debugging purposes

//print_r($table);
//echo $sd_value . "<br>";
//echo $now_value . "<br>";

// Connects to your Database 
// Some database stuff
?>
Click to expand...
Much appreciated, I'm developing my own in Java but interested to compare how php does it.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X