Mweb bloccking Google crawler IP?

M

Mossie22

Banned
Joined
Oct 14, 2013
Messages
5
Reaction score
0
I have just registered a new site with mweb. Been trying to do SEO work, but google cannot have access to my site.
I have done everthing humanly possible from my side, and all indications is that it is getting blocked by the server.
Google's response and even the logfile in cPanel tells me error 403 (no access).
Here the Google rsponse:-
Fetch as Google
This is how Googlebot fetched the page.
URL: http://www.turfscapes.co.za/robot.txt
Date: Sunday, October 13, 2013 at 10:54:51 PM PDT
Googlebot Type: Web
Download Time (in milliseconds): 578
HTTP/1.1 403 Forbidden
Date: Mon, 14 Oct 2013 05:54:55 GMT
Server: Apache
Content-Length: 400
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /robot.txt on this server.</p>
<p>Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.</p>
<hr>
<address>Apache Server at www.turfscapes.co.za Port 80</address>
</body></html>


The next line is from the logs on cPanel:
[Mon Oct 14 07:54:55 2013] [error] [client 66.249.73.206] File does not exist: /home/m7560120/public_html/403.shtml
[Mon Oct 14 07:54:55 2013] [error] [client 66.249.73.206] File does not exist: /home/m7560120/public_html/403.shtml
[Mon Oct 14 07:54:42 2013] [error] [client 66.249.81.196] File does not exist: /home/m7560120/public_html/404.shtml, referer: http://www.google.com/search
[Mon Oct 14 07:54:42 2013] [error] [client 66.249.81.196] File does not exist: /home/m7560120/public_html/robot.txt, referer: http://www.google.com/search
Anybody here can give me some indication where to start looking here please.:confused:

Thanks
Mossie
 
Can you find a robot.txt file at that location? What is it's file permissions?
 
Yes, Robot.txt is just one file as an excample.
rbtext.gif
And, directories permissions is all 755
 
Mossie22 said:
I have just registered a new site with mweb. Been trying to do SEO work, but google cannot have access to my site.
I have done everthing humanly possible from my side, and all indications is that it is getting blocked by the server.
Google's response and even the logfile in cPanel tells me error 403 (no access).
Here the Google rsponse:-
Fetch as Google
This is how Googlebot fetched the page.
URL: http://www.turfscapes.co.za/robot.txt
Date: Sunday, October 13, 2013 at 10:54:51 PM PDT
Googlebot Type: Web
Download Time (in milliseconds): 578
HTTP/1.1 403 Forbidden
Date: Mon, 14 Oct 2013 05:54:55 GMT
Server: Apache
Content-Length: 400
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access /robot.txt on this server.</p>
<p>Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.</p>
<hr>
<address>Apache Server at www.turfscapes.co.za Port 80</address>
</body></html>


The next line is from the logs on cPanel:
[Mon Oct 14 07:54:55 2013] [error] [client 66.249.73.206] File does not exist: /home/m7560120/public_html/403.shtml
[Mon Oct 14 07:54:55 2013] [error] [client 66.249.73.206] File does not exist: /home/m7560120/public_html/403.shtml
[Mon Oct 14 07:54:42 2013] [error] [client 66.249.81.196] File does not exist: /home/m7560120/public_html/404.shtml, referer: http://www.google.com/search
[Mon Oct 14 07:54:42 2013] [error] [client 66.249.81.196] File does not exist: /home/m7560120/public_html/robot.txt, referer: http://www.google.com/search
Anybody here can give me some indication where to start looking here please.:confused:

Thanks
Mossie
Click to expand...

Mossie22 said:
Yes, Robot.txt is just one file as an excample.
View attachment 76067
And, directories permissions is all 755
Click to expand...

Good Morning Mossie22

Please provide me your preferred email address via private message and I will have our hosting team respond to you via email.
 
Mossie22 said:
Yes, Robot.txt is just one file as an excample.
View attachment 76067
And, directories permissions is all 755
Click to expand...

It is my understanding that our hosting team is now in contact with you.

Please let me know if you require any further assistance or feedback ;)

You have a great day!!
 
Yes, I can open the file.
Here is another indication from google regard to my main index.html file:-
"Googlebot couldn't crawl your URL because your server either requires login to access the page, or is blocking Googlebot from accessing your site."
 
User-Agent: *
Disallow:
Basically nothing, as I don't need a robot file.
Want google to crawl the complete site, but used the file for further tests as to check my other html files.
 
Just for the record and to help others that might have same prob,:-

Hi Mossie
Please test again, according to our Unix team this was due to a mod_security ruleset.
Regards
Hosting Team
It's now working fine and all is solved.
Thanks Mweb Guy.
 
Mossie22 said:
Just for the record and to help others that might have same prob,:-

Hi Mossie
Please test again, according to our Unix team this was due to a mod_security ruleset.
Regards
Hosting Team
It's now working fine and all is solved.
Thanks Mweb Guy.
Click to expand...

Awesome, glad we could help ;)

You have a great day!!
 
You might also want to check that robots txt name - you seem to be crawling for ***co.za/robot.txt
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X