Meeting Google’s new Robots.txt guidelines

On October 27th 2014, Google released an announcement regarding changes to their technical webmaster guidelines. The announcement included the following paragraph that really caught the eye of search engine marketers, web designers and online professionals:

“For optimal rendering and indexing, our new guideline specifies that you should allow Googlebot access to the JavaScript, CSS, and image files that your pages use. This provides you optimal rendering and indexing for your site. Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.” (emphasis mine)

Feel free to read the article Google released here. Basically, the above quote boils down to this: Google needs to see the stylesheets and javascript for your site in order to ensure that their idea of what your site looks like is accurate. This way, they are able to place proper emphasis on content that is visually emphasized on the site.

So how do you go about ensuring that your robots.txt file complies with the new guidelines? Simply follow the steps outlined below:

1. Ensure your site has been submitted to Google Webmaster Tools. (https://www.google.com/webmasters/tools/) If not, you’re missing out on a world of helpful feedback regarding your site. Setting up a GWMT account is free, you’ll need to ensure that the site you’re working on in particular has been added to your WMT account and that you have access to it.
2. Run the Robots.txt testing tool:
   (https://www.google.com/webmasters/tools/robots-testing-tool/) WMT will display whatever robots.txt file they have on record for your site.
3. Edit the file to ensure Google has access to your Javascript and CSS files. Editing the file is as simple as clicking in the text window and using it in the same way as any basic text editor.
   
   NOTE: All Users!!!The robots.txt file is basically a guide telling search engines which parts of your website to ignore from their index. As such, it can potentially impact your rankings pretty dramatically. If you’re unsure about what is safe to delete from your robots.txt file, send a link to this post to your webmaster and ask them to make sure that Google has access to your Javascript and CSS files.
   
   NOTE #2: WordPress Users!!! You may notice wp-content and wp-includes listed in your robots.txt. These folders include your site’s stylesheet info and should be removed from the robots.txt file. Simply delete the following rows, if they appear:
   Disallow: /wp-includes/
   Disallow: /wp-content/
4. Click “Submit” under your edited Robots.txt file, and follow the instructions that appear to:

1. 1. Download your updated Robots.txt (self explanatory; the file will be downloaded in .txt format to a location on your hard drive of your choosing)
   2. Upload the new file to your site’s root folder and Check uploaded version. If you’re not hugely technical, you can get your webmaster to tackle this for you, but if you have ftp access you can simply upload the new robots.txt file to your root folder, overwriting the existing file if one exists.
   3. Ask Google to Update their index regarding your new robots.txt file. There is sometimes a delay between you submitting this request and the actual updated robots.txt file being visible in the tester, so don’t be too concerned if you don’t see the changes immediately. Check back in a day or two and you should see your edits.

Staying on top of trends in the world of search engine optimization and online marketing can be very challenging, but doing so has some incredibly rewarding outcomes. Rather than shooting out masses of content, you now have the capacity to produce content targeting your niche, and have the potential customers find you. Your time and effort is devoted directly to those members of the population that are actively looking for the solution you provide. We’ll work with you to determine your online marketing needs, and ensure that you are set up with a solution that yields the greatest possible ROI. So why not get in touch and…

Start being found online!

Guest Author: Jonathan Jenkins is an SEO professional and content writer in Cape Town, South Africa. He enjoys staying on top of the ever-shifting industry he works in, playing soccer, and evenings spent by the grill.