On October 27th 2014, Google released an announcement regarding changes to their technical webmaster guidelines. The announcement included the following paragraph that really caught the eye of search engine marketers, web designers and online professionals:
So how do you go about ensuring that your robots.txt file complies with the new guidelines? Simply follow the steps outlined below:
- Ensure your site has been submitted to Google Webmaster Tools. (https://www.google.com/webmasters/tools/) If not, you’re missing out on a world of helpful feedback regarding your site. Setting up a GWMT account is free, you’ll need to ensure that the site you’re working on in particular has been added to your WMT account and that you have access to it.
- Run the Robots.txt testing tool:
(https://www.google.com/webmasters/tools/robots-testing-tool/) WMT will display whatever robots.txt file they have on record for your site.
NOTE #2: WordPress Users!!! You may notice wp-content and wp-includes listed in your robots.txt. These folders include your site’s stylesheet info and should be removed from the robots.txt file. Simply delete the following rows, if they appear:
- Click “Submit” under your edited Robots.txt file, and follow the instructions that appear to:
- Download your updated Robots.txt (self explanatory; the file will be downloaded in .txt format to a location on your hard drive of your choosing)
- Upload the new file to your site’s root folder and Check uploaded version. If you’re not hugely technical, you can get your webmaster to tackle this for you, but if you have ftp access you can simply upload the new robots.txt file to your root folder, overwriting the existing file if one exists.
- Ask Google to Update their index regarding your new robots.txt file. There is sometimes a delay between you submitting this request and the actual updated robots.txt file being visible in the tester, so don’t be too concerned if you don’t see the changes immediately. Check back in a day or two and you should see your edits.
Staying on top of trends in the world of search engine optimization and online marketing can be very challenging, but doing so has some incredibly rewarding outcomes. Rather than shooting out masses of content, you now have the capacity to produce content targeting your niche, and have the potential customers find you. Your time and effort is devoted directly to those members of the population that are actively looking for the solution you provide. We’ll work with you to determine your online marketing needs, and ensure that you are set up with a solution that yields the greatest possible ROI. So why not get in touch and…
Guest Author: Jonathan Jenkins is an SEO professional and content writer in Cape Town, South Africa. He enjoys staying on top of the ever-shifting industry he works in, playing soccer, and evenings spent by the grill.