Website Dump - Search an returning a range of entries with the related data

R

Ryan619

Active Member
Joined
Mar 20, 2011
Messages
57
Reaction score
0
Hi all,

I have interesting question (at least to me),

I am looking to datamine the FSB website: http://www.fsb.co.za/fais/search_fsp.htm , in order to generate a list of FSPs with their related details pulling through as well.

I would look to search and auto return anything within the range: 1 to 99999999999 for the first field "Search For FSP No"

Thereafter all entries generated need to be dumped with the headings of "FSP No", "FSP Name", "Trading Name", "Fin Year End", "Status".

Also dumped for each entry would be the information pulled from the "Details" button on the right hand side of each entry.

Does anyone have any idea how I would go about doing this?

I would really appreciate any input.

Ryan
 
Ryan619 said:
Hi all,

I have interesting question (at least to me),

I am looking to datamine the FSB website: http://www.fsb.co.za/fais/search_fsp.htm , in order to generate a list of FSPs with their related details pulling through as well.

I would look to search and auto return anything within the range: 1 to 99999999999 for the first field "Search For FSP No"

Thereafter all entries generated need to be dumped with the headings of "FSP No", "FSP Name", "Trading Name", "Fin Year End", "Status".

Also dumped for each entry would be the information pulled from the "Details" button on the right hand side of each entry.

Does anyone have any idea how I would go about doing this?

I would really appreciate any input.

Ryan
Click to expand...

It is possible to programatically fire 99999999999 HTTP requests to the website, each querying a different FSP number, but you'll be doing it for a while and I'm not sure whether the website owners would be appreciative of the additional traffic.
 
The FSB is like GOD in financial services, they will and have not ever replied to my requests.

Not a spam campaign but rather database that can utilized as a reference source for intermediary and acquisition business.

Any help/advice would be appreciated.

Ryan
 
Looks like you can search for partial matches on the other fields.

This should allow you to do 10 searches using 0 - 9 entered in the postal code field, dump each result set into a spreadsheet and remove (possible) duplicates. Voila!
 
I don't know anything about them but:

1. Is it copyrighted (or other rights) data?
2. They could complain that you are attacking and DOS-ing their servers
3. You will need to optimize your logic. From 1 to 99999999999 if 1 request a second is:

99999999999 seconds
1666666666 minutes
27777777 hours
1157407 days
3170 years
 
Thanks WPD, but what I also need is the pulling of data from the Detail button, which pulls through to a separate window with all the company info.
This is the hard part unfortunately.

Thanks for your comment though!
 
That is quite possible, but I am sure that they do have measures in place to prevent the server load you will when mining the data. I think your IP will simply be blocked at some point. Secondly, you could face an investigation and probably prosecution.

In any event, make use of Google to read more about data mining techniques.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X