htaccess restrictions

[CranialBlaze]

1 of my clients sites has become quite popular, unfortunately that is not always a good thing, the site in question is only targeted at South Africans and i did a very good job with SEO so over 1200 unique visitors are coming through daily, but about 40% are from international visitors and with the site being hosted locally that amounts to over 40GB in traffic per month which works out to nearly R10 000 for bandwidth usage.

I went looking for a solution and found a way using htaccess of blocking all ip addresses outside the ZA range, its quite a long list you need to add to the htaccess, the only problem is that search engine bots are now also getting to the 403 Forbidden page, i tested this htaccess on old unused but still indexed site.

<Limit GET HEAD POST>
order deny,allow
# Country: SOUTH AFRICA
# ISO Code: ZA
# Total Networks: 882
# Total Subnets: 15,870,464
allow from 41.0.0.0/11
allow from 41.48.0.0/13
#
deny from all
</Limit>

Above is an extract from the htaccess just to give an idea, looking through logs and searching the net i got the user agent id's for the most important search engines, only problem i do not know how to include it into the file to make sure that it will work.

The main ones seem to be

User-agent: Teoma
User-agent: ia_archiver
User-agent: msnbot
User-agent: Slurp
User-agent: Googlebot

i would greatly appreciate some help fixing up this file and if anyone wants a copy of it they can just pm me, could be very useful if you do not need international traffic and are locally hosted, the less bandwidth you waste on people who are never gonna end up earning you money, the better at the end.

Thanks a lot for any assistance.

 

[guest2013-1]

Google bot doesn't just use their own user-agent for crawling, they spoof several different kinds that looks exactly like a browser sometimes. The best way is to get a list of their IP addresses, however, best to do a reverse DNS and check for googlebot.com, but then you're pushing out guys like Yahoo, Cuil, Bing, Ask etc

If you're interested I can get those IP's for you.

Apologies for not contributing to your original question, but if you look at Web Africa Platinum Linux package that gives you 55gb (locally hosted) bandwidth usage for R600 per month. You also get 5gb web space with it.

R10000 > R600 and then you wouldn't have to worry about the international clients for now.

 

[guest2013-1]

Also, remember the 196 range South Africa has, the 41. range you added will only allow people with ADSL lines to surf to your site...

 

[zagame]

why dont you edit your robots.txt file to exclude crawls and indexing by search engines?

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

should help you on the right track

also

http://www.seoconsultants.com/robots-text-file/

http://www.antezeta.com/blog/search-engine-crawlers - list of all/ most crawlers

hope this is of use

 

[CranialBlaze]

I eventually just added the ip addresses for all the search engines i want indexing, within minutes of updating the htaccess file google was roaming around again.

Google alone eats about 100mb a day just for indexing

 

[zagame]

I eventually just added the ip addresses for all the search engines i want indexing, within minutes of updating the htaccess file google was roaming around again.

Google alone eats about 100mb a day just for indexing

try the robots.txt aswell and see if it makes a difference

 

[CranialBlaze]

try the robots.txt aswell and see if it makes a difference

Not worried about the bots, i have unlimited bandwidth on the current server, the restrictions are for cpu time and script executions, by blocking all nonessential countries that's all fallen way under limit

 

[Chelle]

Not worried about the bots, i have unlimited bandwidth on the current server, the restrictions are for cpu time and script executions, by blocking all nonessential countries that's all fallen way under limit

In your original post you refer to the R10000 per month and looking for a solution and now suddenly it's not a problem

 

[CranialBlaze]

In your original post you refer to the R10000 per month and looking for a solution and now suddenly it's not a problem

The site was moved off a local server to an international. The problem i was referring to had nothing to do with the bandwidth, the question was how to add the search engine bots to the htaccess via their user-agent name, that is the problem here.

 

[guest2013-1]

The site was moved off a local server to an international. The problem i was referring to had nothing to do with the bandwidth, the question was how to add the search engine bots to the htaccess via their user-agent name, that is the problem here.

This is what you said:

1 of my clients sites has become quite popular, unfortunately that is not always a good thing, the site in question is only targeted at South Africans and i did a very good job with SEO so over 1200 unique visitors are coming through daily, but about 40% are from international visitors and with the site being hosted locally that amounts to over 40GB in traffic per month which works out to nearly R10 000 for bandwidth usage.

I went looking for a solution and found a way using htaccess of blocking all ip addresses outside the ZA range, its quite a long list you need to add to the htaccess, the only problem is that search engine bots are now also getting to the 403 Forbidden page, i tested this htaccess on old unused but still indexed site.

If bandwidth isn't the problem then why lead into it with the bandwidth being the problem seeing as though you moved it to an international server? Smoking crack?

Also take note, the reason you have more international visitors to the site than local is because of the server location. I've found, in my short SEO career of only a year, that if the server is located in the US, 60%+ of your traffic will come from the US because the bot's index you higher for rankings in the search engine related to US (google.com as oppose to the rankings you'll see in google.co.za, a locally hosted competitor will always rank higher than you in the .co.za realm)

If you were hosting it locally you'll have an opposite effect.

Please take note of what I said in my response to you regarding the 196 range of IP addresses for South Africa. Even though some of the people have ADSL these days, most go through the nice little invisible proxy and/or dialup as well. So keep that in mind as well seeing as though all IS based traffic will have 196 IP addresses and not 41.

 

[CranialBlaze]

This is what you said:



If bandwidth isn't the problem then why lead into it with the bandwidth being the problem seeing as though you moved it to an international server? Smoking crack?

Also take note, the reason you have more international visitors to the site than local is because of the server location. I've found, in my short SEO career of only a year, that if the server is located in the US, 60%+ of your traffic will come from the US because the bot's index you higher for rankings in the search engine related to US (google.com as oppose to the rankings you'll see in google.co.za, a locally hosted competitor will always rank higher than you in the .co.za realm)

If you were hosting it locally you'll have an opposite effect.

Please take note of what I said in my response to you regarding the 196 range of IP addresses for South Africa. Even though some of the people have ADSL these days, most go through the nice little invisible proxy and/or dialup as well. So keep that in mind as well seeing as though all IS based traffic will have 196 IP addresses and not 41.

The site was moved to an international server 5 days ago because of the bandwidth issue, hence leading with that, the new issue is the system resources. The location of the server has nothing to do with the source of the traffic(i have been doing SEO for 7 years, i monitor and constantly research changes in search engine rules and behaviour), the % of international users has always been between 50% and 60% during the entire 18 months it was hosted locally, during the last month of local hosting the providers had upgraded their server, we had then discovered that the site had always been exceeding bandwidth allocations the provider had just never been notified of it due to the servers logging software or whatever.

After the upgrade we realized it as 3 days into the month we received our first usage warning and starting looking back at the older logs only to realize that because the outdated software was reporting the data usage in byte we had been misreading the usage for the full 18months and what we assumes was 4gb based on the fact that we were not paying close attention and the fact that we were not receiving any usage warnings turned out to be 40GB.

The new server was advertised to essentially have unlimited everything, in the features list their was no specification of CPU and Script Execution Limits.

The only reason we don't move it again was because of all the hassles we had the last time and the fact that we paid a year in advance. a 500mb website with a 100mb database is unfortunately not the simplest thing to move, had to upload the database 4 times before getting a clean import, the instability of local lines makes uploading a 40mb zip file really fun.

So instead of once again going through 2 days of effort to move a site and make the database work and wait for all the local DNS servers to update properly, the simpler solution is to block useless traffic and possibly find a way of allowing certain search engines without having to constantly update the IP list by figuring out how to allow them using their user-agent identifier.

As for the list of ip addresses above, that is a snippet, their are almost 100 lines of ranges just for SA.

As for google and ranking according to TLD, google.co.za is simply a redirect google ranks according to many things, location of the server is not 1 of them. a .com will rank higher than a .co.za if it was hosted in Texas, Europe, Australia or even in SA, international TLD's always rank higher than country specific TLD's, so .biz would be worth more than .co.uk or .co.za and way more than something like .za.net which falls somewhere bellow the bottom of the list.
If you were to build 2 identical sites and host then on the identical servers at the same location and just assign them different IP's or even the same IP and u optimise them in the exact same way, the only difference would be you make 1 a .com or .biz and the other a sub tld like .co.za or .co.us and the .com/.biz will always rank first in a google search.

 

[guest2013-1]

Ok, well I guess we have to agree to disagree with the number of international visits as my experience is what I said it was. But then again, you've been doing it longer than I have

With regards to a server move:

I find it easier to transfer files from server to server (that's providing you have RDP access, and from what I assume, you don't, which means it's a shared enviroment?)

Moving the site is easy, you just point the current DNS to the new IP address. That is usually an almost instant change. Once the IP address has propogated to the rest of SA (which will take an hour or two at the most depending on how lame your ISP is) your server is moved and done with.

Once you can confirm that everything is 100%, you can then put in the request to change the DNS servers on Uniform which would take 24-72 hours to update.

This will ensure that once the update is finished you have no service disruption because by that time you're running on the new server anyway.

The only "downtime" I've ever had was about 10 minutes and that was to do with email routing. Because some servers cache the IP for a specific email server for a while, some emails may be routed to the old server while others go to the new server. I just setup a DNS entry to the old server as well for them to check if there's any old email. 90% of the time it doesn't happen as I do the DNS switch early on a Saturday morning (early as in before 1am)

That's how I switch over servers and I hope you can learn something from it as well. I've done the whole "submit ticket to co.za" thing and they either ****ed it up or I had to wait a long long while.... 3 days is a lot of emails to pull down from the old server and push back up to the new one.... I had .co.za updates

 

[RSkeens]

Moving the site is easy, you just point the current DNS to the new IP address. That is usually an almost instant change. Once the IP address has propogated to the rest of SA (which will take an hour or two at the most depending on how lame your ISP is) your server is moved and done with.

Quite right, just remember to also lower the TTLs or ask for them to be lowered.

 

[guest2013-1]

Quite right, just remember to also lower the TTLs or ask for them to be lowered.

With standard TTL's that shouldn't be a problem, but I've seen South African hosts **** with the TTL's a lot, but good thing to keep in mind.

 

[RSkeens]

With standard TTL's that shouldn't be a problem, but I've seen South African hosts **** with the TTL's a lot, but good thing to keep in mind.

With default TTLs it does make a difference (but it's all relative as to how busy the site is). Each to their own experience though.