The Screaming Frog SEO Spider is a small desktop program you can install locally on your PC, Mac, or Linux machine which spiders websites’ links, images, CSS, script, and apps from an SEO perspective. It fetches key onsite elements for SEO, presents them in tabs by type, and allows you to filter for common SEO issues, or slice-and-dice the data how you see fit by exporting into Excel. You can view, analyse, and filter the crawl data as it’s gathered and updated continuously in the program’s user interface.
The Screaming Frog SEO Spider allows you to quickly crawl, analyse, and audit a site from an onsite SEO perspective. It’s particularly good for analysing medium-to-large sites, where manually checking every page would be extremely labour-intensive (or impossible!) and where you can easily miss a redirect, meta refresh, or duplicate page issue.
The SEO Spider allows you to export key onsite SEO elements (url, page title, meta description, headings etc) to Excel so it can easily be used as a base to make SEO recommendations from.
Screaming Frog SEO Spider is free to use, but imposes certain restrictions. Purchase of a license adds the following functionality:
- The 500 URI crawl limit is removed
- You can access ALL the configuration options
- You can save and re-upload crawls
- You can search for anything in the source code of a website with the custom source code search feature
- You get support for any technical issues with the software
New Features
- Google Search Analytics Integration - You can now connect to the Google Search Analytics API and pull in impression, click, CTR and average position data from your Search Console profile. Alongside Google Analytics integration, this should be valuable for Panda and content audits respectively.
- View and Audit URLs Blocked By Robots.txt - You can now view URLs disallowed by the robots.txt protocol during a crawl. Disallowed URLs will appear with a 'status' as 'Blocked by Robots.txt' and there's a new 'Blocked by Robots.txt' filter under the 'Response Codes' tab, where these can be viewed efficiently.
- GA and GSC Not Matched Report - The 'GA Not Matched' report has been replaced with the new 'GA & GSC Not Matched Report' which now provides consolidated information on URLs discovered via the Google Search Analytics API, as well as the Google Analytics API, but were not found in the crawl. This report can be found under 'reports' in the top level menu and will only populate when you have connected to an API and the crawl has finished.
- Configurable Accept-Language Header - Google introduced local-aware crawl configurations earlier this year for pages believed to adapt content served, based on the request's language and perceived location. This essentially means Googlebot can crawl from different IP addresses around the world and with an Accept-Language HTTP header in the request. Hence, like Googlebot, there are scenarios where you may wish to supply this header to crawl locale-adaptive content, with various language and region pairs. You can already use the proxy configuration to change your IP as well.
Smaller Updates and Fixes
- The Analytics and Search Console tabs have been updated to allow URLs blocked by robots.txt to appear, which we believe to be HTML, based upon file type.
- The maximum number of Google Analytics metrics you can collect from the API has been increased from 20 to 30. Google restrict the API to 10 metrics for each query, so if you select more than 10 metrics (or multiple dimensions), then we will make more queries (and it may take a little longer to receive the data).
- With the introduction of the new 'Accept-Language' configuration, the 'User-Agent' configuration is now under 'Configuration > HTTP Header > User-Agent'.
- We added the 'MJ12Bot' to our list of preconfigured user-agents after a chat with our friends at Majestic.
- Fixed a crash in XPath custom extraction.
- Fixed a crash on start up with Windows Look and Feel and JRE 8 update 60.
- Fixed a bug with character encoding.
- Fixed an issue with Excel file exports, which write numbers with decimal places as strings, rather than numbers.
- Fixed a bug with Google Analytics integration where the use of hostname in some queries was causing 'Selected dimensions and metrics cannot be queried together errors'.
