You probably know this already, since you’re looking for alternatives, but Screaming Frog is a very valuable tool for marketers, especially those looking to audit their website. It’s a search spider, meaning it crawls your website and can harvest a huge amount of data in a relatively short amount of time. A total site crawl will give you a complete readout of your scripts and media, your meta data, your links and their status, and a whole lot more besides. All of that information is incredibly valuable when you’re trying to fix errors and generally make your site look better to the search engines.
The downside, of course, is the price. Screaming Frog has its own free version, but that version is limited in functionality. You can crawl your site for broken links and link errors, page titles and meta data, your meta robots directives, your hreflang attributes, and it can find duplicate content. It can also generate an XML sitemap for you. That said, that’s all it can do, and it has a limit of 500 URLs. For larger and older sites, or for marketers who want more features, you have to go with the paid version.
The paid version gives you additional crawl configuration options, saved crawls, custom source code searching and extraction, integration with Google Analytics and the Search Console as well as Link Metrics. It also renders scripts, lets you set a custom robots.txt file, and gives you tech support. All of that, with an unlimited URL crawl limit, is excellent. It’s also 149 pounds – about $210 at today’s exchange rate – per year. That can be a pretty steep fee.
If you’re experienced in the world of marketing tools, you know that there are likely a bunch of crawler alternatives. You also know that most of them will be limited to only partial functionality, or will be limited in some other way. Let’s go through the various alternatives you have and what features you can access. For each of them.
1. Visual SEO Studio
Visual SEO Studio is first on this list, but don’t make the mistake of thinking it’s the best. The list I’ve put together is not sorted in any particular way; it’s just the order that I found these alternatives and have reviewed them.
VSS has a free “community” version and a paid “professional” version. I’ll mention what the professional version does, but I’ll be focusing on the community version because this post is largely focused on free alternatives rather than other paid alternatives.
Like the free version of Screaming Frog, the free version of VSS is limited to 500 maximum crawlable pages. It’s also limited to a maximum of 2 concurrent connections, meaning it can take a little bit of time to crawl those 500 pages. It will crawl and scrape your page titles and meta data, and will find issues with HTTP and HTTPS. It can find broken links and will inspect your robots.txt directives, as well as generate XML sitemaps for your site, at least in the limited amount it can crawl. It will offer improvement suggestions for meta titles or missing titles, duplication issues, short or overly long titles, missing meta data, and other SEO recommendations. I don’t know offhand whether or not it has been updated for Google’s new longer meta titles, so keep that in mind.
The paid version adds on a specific URL list crawl feature, audits for sitemaps, a health check for Analytics, performance suggestions, a readability analysis, a link inspector, and custom filters. It also has a limit of 150,000 crawled pages, which is massive but still less than some very old websites.
2. Wildshark SEO
Wildshark is a UK-based company making a variety of tools. This one, the SEO Spider tool is most comparable to Screaming Frog and is 100% free. You do, however, need to sign up for their mailing list in order to download the app.
Wildshark’s spider will search your website for missing titles, duplicate titles, and overly short or overly long titles. They do still have the short Google title limit in place, so use your judgment there. They also look for missing or duplicate meta descriptions, which can be quite important and indicate duplicate content issues as well. They’ll even check if your meta description is too long for Google’s results display. They’ll look for missing H tags, duplicate H tags, or overused H tags, as well as pages that have overly long or overly short H tags, which can indicate issues with closing a tag in the code. The tool also looks for broken links, though I don’t believe it checks for redirects or other valid status codes, and it checks for image ALT tags.
The one unique feature of Wildshark is that it will scan your content and let you know if a page has so little content that it might be considered thin content according to Google. It’s not a very sophisticated content scanner – it looks for anything less than 300 words, where I would consider anything less than 1,000 to probably be thin – but it can give you a warning if pages you don’t want visible are showing up.
This is all basic scanning, but sometimes it’s all you need. If you don’t need any more features than the ones I just listed, this can be an excellent tool to use. The primary benefit though, in addition to being free, is that it has no crawl limitations. You can crawl the entirety of a site the size of Forbes if you had the time and the processing power to do it.
3. Spotibo
Spotibo is a basic SEO crawler that, like others before and after it, has a free version with a crawl limit of 500 pages. The primary benefit, however, is that it’s a web app rather than a downloadable program. This means you can use it on any platform, be it Windows, Mac, or another architecture. It won’t freeze up your computer using up all available processing power on the crawl.
Unlike other tools, Spotibo’s 500 URL limit is not a lifetime limit of the app. It’s actually a monthly limit. If you have several small sites you want to crawl, you can stagger them throughout the year and keep an accurate rotating overview of your crawl data without paying a penny. They do have a limit of three websites, however.
As far as what it crawls, it’s fairly standard. It will crawl your page and scan links, reporting their status codes. It looks for server status codes for your own pages as well, and will show redirects, 404s, and other errors. It looks for content issues, including the usual URL structure scans, meta titles, and meta description issues. It also scans images looking for their URLs, titles, and alt text. You can see, filter, and manipulate your data all within their dashboard rather than needing to export it to filter through it as well.
4. Xenu’s Link Sleuth
This option likely made some of you recoil just upon clicking the link, and that’s completely understandable. Xenu’s grasp on web design is straight out of 1985. It’s very simple, very basic, and very utilitarian. The latter also describes the program you’ll be downloading.
Link Sleuth is a very simple program without a lot of various options, no sugar-coating, no analysis, and no fancy dashboards. You basically point it at a domain and let it go, and it presents you with an organized list of every link on your page and the status of those links. Your final crawl report will show the URL of the link, the server status code of that link – with associated color coding for the entire entry – the type of link or page if applicable, the size of the page if applicable, and the meta title of the link. There are a few other fields as well.
Xenu is a strange guy, and this app is only a side project. As such, the most recently updated version is from 2010. It supports SSL pages, and can detect and report redirects, but don’t expect it to do anything extremely sophisticated. It’s also not really limited; you point it at a domain and let it run, and it will happily eat up both your computer’s available processing power and your server’s available bandwidth if necessary.
The app is completely free and unlimited, and it’s so small it downloads in a few seconds on any modern broadband connection. On the other hand, it’s very limited. All it does is scan your links and report their status and information. That’s it, that’s all, end of app. It’s very good at what it does, but it doesn’t do a whole heck of a lot.
5. Scrutiny
An app developed by Peacock Media, Scrutiny is a Mac program rather than a Windows-focused crawler like so many other options. It’s one of the best options for Mac SEOs these days. It has a free trial and a one-time purchase, which costs $115.
The app will do link checking in a way that’s very similar to Xenu or Screaming Frog. It crawls your pages and looks for links and their statuses, and it can even log into members sections or scan links inside PDF files, as well as render scripts to show content before crawling.
For SEO scanning, Scrutiny looks at meta data in both the title and description, as well as image alt text. It also scans content and looks for potential content issues, like keyword stuffing, thin content, duplicate content, and overly mixed content. It can also export an XML sitemap for you. Additionally, the app can run your site through a spelling and grammar check, in case you’re concerned about such issues. It’s not perfect, but hey, it’s a unique feature other crawlers don’t offer.
So what’s the limitation of the free version? Rather than a crawl limit, the app simply works for 30 days before shutting itself down. You can pay for the license then, or you can move on to another app if you don’t like it. You get some updates, but I’m not sure if you get lifetime updates with the paid program – they have some indications on their site that a major version update will cost an additional fee of about $20. Either way, it’s still cheaper than Screaming Frog and a lot of paid alternatives, and it’s a one-time fee. Additionally, with the free version, exports are disabled.
6. Link Assistant’s Website Auditor
Another of the better options, this one also has a limited free version. You have a cap of 500 URLs as the standard, for both analysis and overall statistics. It can make you a sitemap and manage your robots.txt file for you.
As far as what it scans for, there’s a lot going on. You get a full HTML scan looking for errors and violations of best practices. You get a full indexation analysis. You get duplicate content scans and optimization options for your URL structure. You get social media monitoring to a certain extent, and you get a content analysis for SEO-friendly writing. You can even get some competitive analysis out of the app. It supports “up to 571 search engines” though, of course, only a few of those actually matter. You get SERPs reporting, branded reports, custom templates, and workspace filters. It works through a proxy server connection, on any operating system, and updates automatically.
On the other hand, you do not get data exports, Dropbox storage, or cloud storage. You can’t run scheduled tasks, print out reports, or email reports. Basically, you’re limited to accessing your data directly through the app. That’s not a bad thing, though; the app has a lot of good filters and display options for showing you what you need to see.
7. Honorable Mentions
There are a few other options you can look into, if you’re willing to be a bit flexible on your pricing.
- SERPStat is a great app, but it’s limited to 30 queries for the free scan before you need to buy a plan.
- DeepCrawl has a very robust scanner and has a lot of extra features, but costs you $80 per month to start.
- WebBee is a free tool that does a lot of what Screaming Frog will scan, but has a few limitations, including a lack of technical support and only a single machine license.
- BeamUsUp is free and robust, but is only available for Windows. It does URL and meta data scans, duplicate content scans, and exports to Excel or Drive.
Of course, Screaming Frog is still the gold standard. I really recommend saving up for it if you possibly can.
