SEO Crawlers Tools21
This is a shortlist of SEO Crawlers Tools. This list is manually curated and frequently updated by ex-Google engineer Fili.
HTTP Status Tester
Check from different countries around the world the HTTP status code, HTTP response, HTTP headers, redirect chains and applicable robots.txt rules for multiple URLs.
Lumar, previously known as DeepCrawl, is the world's most comprehensive website crawler. Analyse your website architecture to understand and monitor technical issues to improve SEO performance.
Professional crawling tool for on-page, content and structural analysis.
Desktop crawler without limits.
Real-time SEO auditing and content tracking.
Crawly spiders and extracts complete structured data from an entire website.
Forecheck crawls your site to uncover issues that may be holding you back from Google’s first page
HeadMasterSEO can quickly check URLs in bulk for status code, redirect details, response time, response headers and HTTP header fields.
JetOctopus is a website crawler and auditor that helps you get valuable on-page data on every URL and improve SEO performance.
OnCrawl is a SEO crawler and log analyzer that helps you improve your SEO performance.
Online SEO Crawler
A fast and flexible SEO website crawler to help identify technical SEO issues.
Ryte is an all-in-one software for strategic website quality management that monitors, analyze and optimizes the key elements of your website.
Screaming Frog SEO Spider
The SEO Spider is a desktop website crawler and auditor for PC, Mac or Linux which spiders websites’ links, images, CSS, script and apps like a search engine to evaluate onsite SEO.
Seobility is a online SEO crawler for better website optimization.
Crawl and analyze your website.
SEOmator is somewhere between full-scale crawlers and SEO graders and crawls up to 1000 pages from the website to provide identify SEO issues.
SEOSpyder is a crawling software for Mac OSX that provides its users with in-depth SEO data and features of a particular website.
Sitebulb is a desktop website crawler and a tool for SEO auditing.
SiteCondor programmatically scans your website and extracts important site information which is then stored for your reference.
WebSite Auditor checks websites' internal health, reporting on structural, coding and linking defects for every webpage.
Xenu Link Sleuth
The classic Xenu’s Link Sleuth checks Web sites for broken links.
Your tool listed here?
Do you know of a digital marketing tool currently not listed?