Google recently unleashed a shiny new web crawler, “GoogleOther,” designed to give its primary search index crawler, Googlebot, a much-needed break.
According to Google Analyst Gary Illyes, the new crawler will handle non-essential tasks like research and development (R&D) crawls, allowing Googlebot to focus on its primary job of indexing the web.
The update is anticipated to help the tech giant streamline and optimize its web crawling operations.
While Illyes reassured the new crawler won’t significantly impact websites, it begs the question: What does this mean for search engine optimization (SEO) efforts?
In this blog, we’ll explore GoogleOther’s limitations, its possible effect on search engine rankings and whether or not businesses should be concerned.
A Quick Background on Web Crawlers, User Agents and Googlebot
To truly understand how GoogleOther updates affect web crawling, it’s important first to review the basics of web crawlers, Googlebot user agents and the role of Googlebot in the web crawling process.
Google Web Crawlers and User Agents
Web crawlers, also known as robots or search engine spiders, systematically discover and scan websites by following links from one page to another. Search engines use Google spiders to gather information about web pages and curate recommendations for search queries.
To identify themselves to servers, Google web crawlers use a user agent, a string of text included in the request headers sent to the server.
The Googlebot user agent then tells the server which bot requests to crawl the page. This enables website owners to monitor bot activity and limit Google crawling access if necessary.
The server responds with a status code indicating whether or not Google spiders are permitted to crawl the website.
Googlebot and Rankings
If Google crawling is allowed, Googlebot will analyze the web page, including text, images and links. These pages are ranked by relevance, with the highest-ranked pages considered the most relevant to the query.
This search results ranking is based on an algorithm considering various factors such as keywords, content and backlinks from reputable sites.
To improve a website’s ranking, many businesses hire a technical SEO consultant for their website optimization services or on page optimization services.
Where GoogleOther Comes Into Play
The web crawling process is continuous, with Googlebot visiting and re-visiting websites to ensure the Google search index is up-to-date with the latest information.
However, with billions of pages to be indexed, you can imagine how resource-intensive this task can be. Google web crawlers like Googlebot must adapt to handle the increasing amount of data efficiently.
With GoogleOther on board, Google can alleviate some of the strain on Googlebot search engine spiders by assigning non-essential tasks to the new crawler.
Splitting Responsibilities Between Googlebot & GoogleOther
GoogleOther will primarily be used by Google’s product teams for internally building the Google search index. As Illyes stated on LinkedIn:
We added a new crawler, GoogleOther to our list of crawlers that ultimately will take some strain off of Googlebot. This is a no-op change for you, but it’s interesting nonetheless I reckon.
As we optimize how and what Googlebot crawls, one thing we wanted to ensure is that Googlebot’s crawl jobs are only used internally for building the index that’s used by Search. For this we added a new crawler, GoogleOther, that will replace some of Googlebot’s other jobs like R&D crawls to free up some crawl capacity for Googlebot.”
Essentially, it will take over various tasks historically owned by Googlebot, including research and development (R & D) crawl. And by historical, here’s what we mean:
GoogleOther’s Limitations and Features
GoogleOther inherited Googlebot’s infrastructure, which means it possesses the same limitations and features when crawling web pages. This includes:
• Host Load Limitations: Subject to the same limits on how much load it can generate on a server, preventing it from overwhelming a site’s resources or causing downtime.
• Robots.txt Restrictions: Obeys the same robots.txt rules as Googlebot search engine spiders, but with a different Googlebot user agent token. This allows site owners to control which parts of their site are crawled and which are not.
• HTTP Protocol Version: Uses the same HTTP version as Googlebot, currently HTTP/1.1 and HTTP/2 (if supported by the site).
• Fetch Size Limit: Subject to the same page size limit as Googlebot, currently set at 10MB. This prevents large pages from consuming excessive resources, which could slow down the Google crawling process.
As Ilyes pointed out, GoogleOther is basically Googlebot under a different name.
What GoogleOther Means for Your SEO Strategy: Experts Weigh In
While Google has assured webmasters that the new crawler won’t have a significant impact on websites, many SEO experts are still wondering about its potential effects on site rankings
Ronnel Viloria, Lead Technical SEO Consultant at Thrive, said it is too early to determine how GoogleOther will impact SEO efforts. Given that GoogleOther is a recent addition, no case studies are available to indicate how it may influence rankings and traffic.
“Google regularly updates its algorithms and crawlers, and these changes can impact search results and website rankings. However, until more information is available about GoogleOther, it’s impossible to predict how it might impact SEO,” Viloria said.
When it comes to optimizing your SEO campaign strategy, Ronnel advised sticking with your current systems while keeping an eye on this new update. However, if your progress has been stagnant or slow, he recommended exploring new opportunities to rank well and attract more visitors to your site.
“One thing I’m sure of is that you should continue to focus on creating high-quality content relevant to your target users. This is your best bet to rank higher in Google SERPs and attract more audiences,” he added.
In addition to high-quality content, on page optimization services, such as keyword research, title and meta tags optimization, image optimization and internal linking can also significantly improve your website’s SEO performance.
These services help search engines better understand your website’s content and context, making indexing and ranking your pages for relevant queries easier.
How To Monitor GoogleOther
If you’re still feeling wary about GoogleOther, here are some steps to incorporate into your SEO campaign strategy to monitor its crawling activities:
• Review Server Logs: Regularly monitor server logs to identify GoogleOther requests. This helps you understand its crawling behavior and the pages it visits.
• Keep robots.txt File Updated: Make sure your robots.txt file is updated with special instructions for GoogleOther so you can control how it crawls your page.
• Monitor Google Search Console (GSC) Crawl Stats: Use GSC to track changes in crawl frequency, quantity, budget or number of indexed pages since GoogleOther was implemented.
• Track Website Performance: Monitor web performance indicators, such as bounce rates, load times and user engagement to spot issues that could manifest as a result of GoogleOther crawling web pages.
Adapt to GoogleOther Changes With Ease
As “GoogleOther” continues to raise questions about its potential impact on SEO, website owners and businesses can take proactive steps to stay ahead. One such step is to partner with top SEO experts like Thrive.
By partnering with Thrive Internet Marketing Agency, businesses can gain a competitive advantage and adapt quickly to any changes in the SEO landscape.
Our technical SEO consultant will help optimize your website content, build high-quality backlinks, conduct comprehensive keyword research and much more to improve your rankings.
Contact Thrive today to see how our website optimization services can help you stay prepared for any potential impact from GoogleOther.