Google not indexing my site

D

dal2000

Expert Member
Joined
Oct 7, 2008
Messages
1,709
Reaction score
0
Location
KZN
I'm about to lose it with google. Google keeps saying that all my pages are being blocked by robots.txt when I look at webmaster tools. Now I have changed the robots.txt file three times and every time it says blocked by robots.txt. allow /, no allow, whatever I try doesn't seem to work. I do the "test" and it says all robots allowed, but google just won't index my site and keeps saying blocked by robots.txt. Anyone got any ideas? :(
 
If it says blocked by robots.txt it's being blocked by robots.txt

Soooooooooooooooooooooooooooooo my suggestion would be to delete robots.txt and see how you go from there... if it still finds a robots.txt file you've been editing the wrong file yo
 
AcidRaZor said:
If it says blocked by robots.txt it's being blocked by robots.txt

Soooooooooooooooooooooooooooooo my suggestion would be to delete robots.txt and see how you go from there... if it still finds a robots.txt file you've been editing the wrong file yo
Click to expand...

I LoL'ed when reading that
 
dal2000 said:
I'm about to lose it with google. Google keeps saying that all my pages are being blocked by robots.txt when I look at webmaster tools. Now I have changed the robots.txt file three times and every time it says blocked by robots.txt. allow /, no allow, whatever I try doesn't seem to work. I do the "test" and it says all robots allowed, but google just won't index my site and keeps saying blocked by robots.txt. Anyone got any ideas? :(
Click to expand...

As mentioned, delete robots.txt and take it from there. Status on Webmaster Tools won't be immediate so give it a while.
 
My site is using wordpress as a platform so basically it's a virtual robots.txt. I had it set to private and then set it to public. One week later google still says blocked, thats when I created a "real" robots.txt to allow all. Even with this "real" robots, it's still saying blocked. Status of the pages blocked was updated today, so it's not "out of date".

In the webmaster tools google is showing the updated version of the robots file. Doesn't make much sense...
 
If your robots.txt is @ http://www.leopardwalklodge.co.za/robots.txt then it is wrong - remove the line in bold:
User-agent: *
Disallow:

To get back on your feet, create a sitemap (go to www.sitemap.org to read up on it) and include it in your robots.txt like so:

User-agent: *
Sitemap: http://myhost.co.za/sitemap.xml

BTW: A mistake like this costs you dearly, it will take a few weeks to get back into the index again. The sitemap will help to speed up things.
 
Last edited:
maybe it's choking on the gzipped sitemap.
did you try removing robots.txt altogether?
 
The other reason could be that your domain was banned before you bought it. However, I see you have 1 page indexed in Google... maybe if you got rid of all the "Coming Soon" stuff and actually add content you'll see a lot more happening
 
i don't see any evidence robots.txt being auto-generated on my wp blog, or any robots.txt on my vanilla wp source, so i presume an seo plugin is generating it.
assuming that if he deletes it via ftp it regenerates...
 
Acid it says all my pages are being blocked by robots.txt. Surely google won't say that if there is an issue with content?

Murray I had no robots.txt file. Wordpress was generating the virtual one, I told wordpress to allow search engines and it still said blocked. That's why I created the "real" file, to allow google. Made it with google webmaster tools too
 
Last edited:
AcidRaZor said:
The other reason could be that your domain was banned before you bought it.
Click to expand...

That also happens - he can request a recheck via Webmaster Tools if needed. I did this and my domain (which was previously banned by some idiot) was indexed again and I never had problems again.
 
dal2000 said:
Magicdude thats not the site, view the robots.txt here http://sa-accommodation4u.com/robots.txt
Click to expand...

I revert back on my previous statement. "Disallow:" is actually valid. But if one has nothing else in the robots.txt, then just get rid of it completely. @Dal2000: Your sitemap is not very good - you have URLs such as "http://sa-accommodation4u.com/order-cancelled/" in there, which don't really have any SEO value. Something which will work for you (at bidorbuy we do something similar) is to produce sitemaps based on categories (or in your case you would create a SiteMapIndex.xml for your main site which includes individual sitemaps for your property locations/regions).

With all the optimisations we have done, we are now getting 160,000 hits per day on sitemap-xml crawled and 10hits/sec via GoogleBot - which is quite good. On another note, I suggest to install YSlow, Pagespeed and HTML Validator for Firefox:
- YSlow reports many issues and your site is rated E (57 out of 100 points) - you should easily achieve above 80
- PageSpeed (which GoogleBot uses and also shows up in Webmaster Toolkit) gives you 70/100 (i.e. 404s on /wp-content/themes/directorypress/themes/Blue2/images/background.gif)
- HTML validator reports over 100 errors.
- Look at www.webpagetest.org to test your speed for performace - not very good stats

Just looking at your HTML, jquery.js is included multiple times, lots of invalid HTML/unclosed tags etc - this will not affect the crawling, but will negatively affect PageSpeed as well as user-experience.

Google Webmaster Toolkit will report most of these issues. In the Sitemap section you will be able to see any errors on the sitemap, but I doubt that you get a high crawl-rate on the sitemap due to the content listed.

I also suggest that while you have no listings in regions, to provide some quality content - i.e. the link "/category/kwazulu-natal/amanzimtoti/" does not give the user anything and GoogleBot will penalise you for this. You will probably
see this under the Crawl errors (i.e. duplicate title-tags/too short title/meta info).

BTW - Google has crawled your site, but just not indexed anything of value (look here: http://www.google.co.za/search?q=si...hs=RO&rls=org.mozilla:en-US:official&filter=0). My suggestion:
Populate the sitemap with proper content (it is now too late to remove it, and the only way you get it right is to fix the sitemap and resubmit - if done right, you will be able to fix it within 10 days).

You also have no inbound links (i.e. http://www.google.co.za/search?hl=e...commodation4u.com&aq=f&aqi=&aql=&oq=&gs_rfai=). Try and get some quality referrals from authoritative sources (government or universities are truly best).
 
Last edited:
Thanks magic. Looks like I have a truck load of work to do...
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X