What to do if you suddenly can’t find yourself on a known search page in Google?
It can happen to the best of sites. Above all, if you find your website disappears from Google – Don’t Panic! Remember, your website may disappear very briefly when Google is busy re-indexing pages from it. This shouldn’t last for long, and if you’re site hasn’t resurfaced 24 hours later, consult the Just Roger IT! Manchester SEO Blog’s essential checklist of common issues!
Problems With The Web Page?
First, check there are no issues with the HTML in the page you expected to see listed. Is there any malware lurking? Has someone posted excessively spammy comments or links? Do the pages exist, or have they been renamed or moved? Make sure everything is present and correct!
Problems With Robots.txt
Make sure you have not disallowed anything in robots.txt, or have anything that might confuse spiders. Generally I only ever recommend:
Allow /
Sitemap : <your sitemap location>
Get Your Head Examined
Review the <head> tag carefully (there are a number of exciting ways you can shoot yourself in the foot here!) Check your META Tags for any ‘robots’ directives. Google, Yahoo and Bing will not read your page (or recommended for search) if you have ‘noindex’ set. Generally, I would avoid anything robot related, and leave them to make up their minds, but that’s just me! Ensure there are no tags such as ‘pragma’, ‘no-cache’, ‘no-store’ or ‘must-revalidate’ as I have seen these cause problems. In fact – avoid any META Tags you are unsure of!
Problems With Server?
Is the server behaving sensibly? Try accessing the page, does it load quickly? Most bots will have a timeout, after which time they’ll just assume your server is offline and move on to their next crawl destination. Remember, they’re tremendously busy people. Despite what some people say, good and reliable hosting is essential for good listings. Sam Spade is a great tool for checking exactly what the server is sending out. To the Web and have a good read. Pay especially close attention to the HTTP headers. There are some directives such as ‘pragma’ with either ‘no-cache’, ‘no-index’ and ‘must revalidate’ that can cause issues in some cases. This is especially true if there is a (silly expiry date in the header such as 1st of January 1900!)
Check Your Backlinks
Have a look to see what type of links you find linking to your site. Are there any odd ones that have appeared suddenly? Has the number gone up or down suddenly? If Google believes some unethical link-building has occurred it may penalise. Anything you are unsure about, investigate!
Domain Responsibility Issues
A very subtle bogeyman, is if you have something that resembles a sub-domain such as mysite.uk.com, then Google suddenly decides uk.com is public enemy number 1 because of some intense spamming of maleware at blackhat.uk.com, so that anything .uk.com suddenly plummets. For this reason I would suggest avoiding these, particularly .gb.com domains where this has already happened.
Still Struggling for Ideas?
If you have read this far, and still not sure what is causing it, a great tactic is to install Google Webmaster Tools. Verify your site so you have access to the full Diagnostics and specifically the crawl errors section. Try to resolve any 404 file not founds or anything other anomalies that Google draws to your attention. I once re-configured my server so it never gave error 404 not found, but instead gave 200 OK and tried to guess at the most likely page based on name. Needless to say Google left a polite notice saying it had noticed. Better not anger the Google Gods!
In general, think back to what was last changed. Stay calm, work methodically and eliminate the variables!
Tags: find, Google, robots.txt, search engine optimisation, SEO, website
[...] Essential SEO Checklist To Finding Yourself http://just.roger-it.co.uk/my-.....om-google/ [...]