SEO Crawlers Tools
21This is a shortlist of SEO Crawlers Tools. This list is manually curated and frequently updated by ex-Google engineer Fili.
HTTP Status Tester
Check from different countries around the world the HTTP status code, HTTP response, HTTP headers, redirect chains and applicable robots.txt rules for multiple URLs.
Lumar
Lumar, previously known as DeepCrawl, is the world's most comprehensive website crawler. Analyse your website architecture to understand and monitor technical issues to improve SEO performance.
Audisto
Professional crawling tool for on-page, content and structural analysis.
BeamUsUp
Desktop crawler without limits.
ContentKing
Real-time SEO auditing and content tracking.
Crawly
Crawly spiders and extracts complete structured data from an entire website.
Forecheck
Forecheck crawls your site to uncover issues that may be holding you back from Google’s first page
HeadMasterSEO
HeadMasterSEO can quickly check URLs in bulk for status code, redirect details, response time, response headers and HTTP header fields.
JetOctopus
JetOctopus is a website crawler and auditor that helps you get valuable on-page data on every URL and improve SEO performance.
OnCrawl
OnCrawl is a SEO crawler and log analyzer that helps you improve your SEO performance.
Online SEO Crawler
A fast and flexible SEO website crawler to help identify technical SEO issues.
Ryte
Ryte is an all-in-one software for strategic website quality management that monitors, analyze and optimizes the key elements of your website.
Screaming Frog SEO Spider
The SEO Spider is a desktop website crawler and auditor for PC, Mac or Linux which spiders websites’ links, images, CSS, script and apps like a search engine to evaluate onsite SEO.
Seobility
Seobility is a online SEO crawler for better website optimization.
SEOcrawler
Crawl and analyze your website.
SEOmator
SEOmator is somewhere between full-scale crawlers and SEO graders and crawls up to 1000 pages from the website to provide identify SEO issues.
SEOSpyder
SEOSpyder is a crawling software for Mac OSX that provides its users with in-depth SEO data and features of a particular website.
Sitebulb
Sitebulb is a desktop website crawler and a tool for SEO auditing.
SiteCondor
SiteCondor programmatically scans your website and extracts important site information which is then stored for your reference.
Website Auditor
WebSite Auditor checks websites' internal health, reporting on structural, coding and linking defects for every webpage.
Xenu Link Sleuth
The classic Xenu’s Link Sleuth checks Web sites for broken links.
Your tool listed here?
Do you know of a digital marketing tool currently not listed?