Why did my Technical audit fail with crawlability issues? – Gumshoe

A Failed status with crawlability issues means our crawler tried to load your page and could not complete a clean read. This is about whether an automated visitor can reach and load your page. It is not a reflection of your content quality or your AI Optimization Score. When the crawl cannot complete, there is nothing to score, so the run is marked Failed.

The good news is that the results give you everything you need to find the cause, and it almost always lives on the website’s side: a redirect, a security or bot rule, or a server error.

What the crawlability issues list is telling you

Each entry in the list is one failed attempt to load your page. Every attempt shows the time it happened, a short description of what went wrong, and, where available, the HTTP status code and the time the server took to respond. Those three details are your starting point for tracking down the cause on your site.

A very fast failure usually points to an outright block or redirect. A very slow one that still fails often points to a timeout or an overloaded server.

What the status codes tell you

The HTTP status code next to each attempt tells you how your server, or a service in front of it like a CDN or firewall, responded to our crawler:

• 200's (success): The server returned a response, but it was not usable as a full page. This often means the crawler received a near-empty shell or a bot verification or challenge page instead of your real content.
• 300's (redirect): Your server sent our crawler to a different URL. If the page redirects to a regional site, a login, or another page, we may never reach the page you intended. Region and language redirects are a common cause here.
• 400's (request refused or not found): 401 and 403 mean access was denied, often by a firewall, bot protection, or a login requirement. 404 means nothing was found at that URL. 429 means the server rate limited us for too many requests.
• 500's (server error): Your server, or a CDN, proxy, or load balancer in front of it, hit an error while responding. 500 is a general server error, and 502, 503, and 504 usually indicate a gateway, availability, or timeout issue. These are frequently intermittent.

Why a page that loads fine for you can still fail

Automated requests are treated differently from those made by a person clicking through in a browser. Firewalls, bot protection, and region-based redirects often wave a normal visitor through while blocking or redirecting an automated crawler. That is why a page can look perfectly fine to you and still fail the audit. It is also why results sometimes vary from one run to the next: the protection triggers on some requests and not others.

How to find and fix the cause

1. Open the failed run from the Analysis results screen and note the exact time and status code for each attempt.
2. Confirm the page loads publicly in a normal browser, and that the URL you entered is the final destination rather than one that redirects somewhere else.
3. Match those timestamps against your own server, CDN, or firewall logs to find the exact requests that failed. The status code tells you where to look: redirect rules for 300's, access and bot rules for 400's, and server or gateway health for 500's.
4. Address the issue on your side (adjust the redirect, review the security or bot rule, or resolve the server error), then republish the page if you changed anything.
5. Return to the Technical audit, open the page, and click Reanalyze to run a fresh scan.