Why Google Isn’t Crawling Your Dealership’s Website

Featured Video Play Icon
Posted on by Kyle Murray
Categories: SEO Tagged: , , , , , , , , , , , , ,

So, you have a webpage that’s not indexed by Google…that’s a problem. So let’s troubleshoot that.

Well, unless you have something more important to do.

(Newsflash…you don’t)

If Your Page Isn’t Currently Indexed
One issue could be that you accidentally told Google not to index your site. How? Well, there are two ways this can happen. One is you have a robots.txt file on your site, and it’s instructing Google to stay away. To find your robots.txt navigate to https://your-website.com/robots.txt

Fair Warning: at this point things can get a little technical so we are just going to graze the surface. Your website is broken down into folders. And your robots.txt file will declare rules that allow or disallow crawlability of a folder or specific page for a specific user-agent. A user-agent is what identifies what type of user it is.

For example:
User-agent: googlebot
Disallow: /admin/

This would block users with user-agent “googlebot” (which is what Google uses) from crawling any page within the /admin/ directory.

So, Here’s What to Look For
If there is anything that has “disallow” that shouldn’t have it under the user agents * (which means all user agents) or googlebot, then it’s being blocked by Google. If you are not sure about any of this, make sure to contact your website administrator.

The other way is you can have a meta tag that says “noindex” or “nofollow”.
“noindex” means “don’t save information about this page”
“nofollow” means “don’t travel to any links from this page”.

How Can You Check This on Your Page?
Once you’re on that page right click somewhere and select “view page source”.
Remember, right click on the page itself. Don’t click on a link or a button. It wont work.

Once you’re on the page source press CTRL F and type inThis will automatically highlight, and it will take you to the meta tag if you have one. Check the content section of the meta tag. It will say content=””. What you don’t want to see in the content is “noindex” or “nofollow”. If you see either of those, contact your website admin and have them fix it.

Remember Last Week…
…when we talked about submitting sitemaps? Make sure to check that again to see if Google is recognizing your sitemap.

There is a possibility that the page that can’t be crawled properly due to the structure of your site. What does that mean? Well, websites are usually structured so you land on the homepage. From there you should be able to navigate to each page from links or buttons in a chainlike pattern. If for some reason you have a page that is “floating” or can’t be reached naturally by clicking those links, there’s a good chance that Google can’t crawl it. If this is the case contact your website administrator and find a natural place to add in a link to the floating page.

It Could Even Come Down to Page Speed
As my friend Aaron would tell you, page speed plays a part in Google’s ability to index your site. If your site takes a loonnnggg time to load then Google will give up and move on. You need to make sure that your site is at least quick enough for Google to read, because chances are if Google won’t wait that long your potential car buying customers won’t either.

There you have it, a multitude of ways to troubleshoot why your page isn’t properly being indexed by Google.