The Agency’s first 48 hours’ findings.

The first step that a good on page SEO agency does is to distinguish between what is a signal and what is noise. Most crawl reports will create dozens of flagged issues, and the ‘sense of ownership’ a team has of the site is to want to correct all of them. The first step the agency took was the reverse: prioritisation in order to find out what problems were hindering Google’s ability to index and rank the pages that really mattered.

Three was the number of critical categories that came back.

This was the first one – server response inconsistency. Two high-value service pages (and the blog index) were periodically returning 503 responses to Googlebot, which was a subset of key pages. The website performed well on desktop for users. However, Google’s server logs revealed that Googlebot was hitting the site during times when a third-party integration was causing response times to become overloaded, and it was also turning away from crawling those pages altogether. It’s been going on for 11 weeks.

The second was redirected chains. Once migrated the CMS last year, some internal links remained that were 301 redirected to the new URLs. Some of those redirected URLs had been redirected yet again. This led to three- and four-hop redirect chains, which caused Google to crawl more slowly through the parts of the site that had such chains. This caused Google to crawl more slowly through parts of the site that had three- and four-hop redirect chains, and in a few cases, it even ceased crawling through those parts.

The third problem was due to some robots.txt misconfiguration. A temporary Disallow rule had been added during a staging period, and it was never taken off after launch. That policy was preventing Google from indexing an entire subfolder where Google had fourteen blog posts that were on the topics that the business had been working on for months and months.

How these issues have been overlooked.Why these issues were not identified.

The next question after the audit results was quite simple: “Why wasn’t this identified earlier?”

The truth was that the team had been watching the rankings and traffic but not the technical aspects that sit below it. The urge has been to check content, competitors, and algorithm changes, instead of checking if Google is visiting the pages when rankings suffered. There were tools at hand. The page indexing report was displayed on Google Search Console. However, if there wasn’t anyone who knew specifically what to look for and how to interpret the signals, the signals would have been in plain sight and still not been “seen.

An experienced on-page SEO company fills this hole in no time. The Google Search Console data, combined with the data from a dedicated crawler such as Screaming Frog and server log analysis, provides a trained auditor with a full view of what Google is seeing on a site that none of these metrics on their own can provide.

What is fixed and what is not, as well as the order of the fixing.

The server response problem had been the top priority due to its being the most difficult blocker. The agency worked together with the development team to determine what the integration would be during peak crawl times and pushed that to a background queue, and, after forty-eight hours, noticed that Googlebot was getting consistent 200 responses.

Next, the redirect chains were taken care of. The agency created a complete inventory of all internal links on the website that reference non-final URLs and gave the agency new link targets for each of these. The development team updated them across templates and static pages, with a single deployment. Three- and four-hop chains turned into direct links to the end stations.

It only took a couple of minutes to run the robots.txt fix, and a couple of weeks for it to take effect. The Disallow rule was removed, pages with the affected posts were added to Google Search Console, and after 14 days, Google had crawled and indexed the 14 blocked blog posts.

The Rankings Recovery Timeline

Crawl recovery doesn’t happen overnight. Google has to take some time to recrawl the affected pages, re-evaluate their content, and re-rank pages previously held back due to technical reasons.

In this instance, the first noticeable change was in the Search Console impressions data after just three weeks of the server update. In weeks 4–6, rankings for the impacted service pages started to rebound. In the next eight weeks, the blog posts were indexed and started to show up in search results and began to rise in rank by relevance.

At the 11-week mark, traffic of any kind had returned to pre-decline levels. Some pages that were crawlable again and had fresher indexing had better rankings than they had previously due to the structural cleanup also affecting how Google interpreted the site as a whole.

This process gave us a lot to learn about on-page SEO.

The biggest learning that this experience taught me is that on-page optimization is not just about what’s on the page. All these elements (including the tags, content, headings, and structured data) have relevance, but only when Google can reliably find the page. Accessibility is built on the foundation of technical accessibility.

Also, it reiterated the fact that crawl errors do not always make a loud noise. They are not necessarily going to generate dramatic error messages or site failures. The worst ones are quiet: those that take a little too long to respond to a crawl window; those that are in a redirect chain that is one hop at a time too long for a crawl window; or those that are a robots.txt rule that is never supposed to be live. They all build up without any notice, and the rankings slowly slip away.

Conclusion

Crawl errors impact a business, but are not a business problem. If key pages are not accessible by Google, just creating more content and links does not help with the rankings. Once the access issue is resolved, they recover.

What the experience showed is that an on-page SEO agency that has the technical expertise does something that the in-house teams, who are monitoring output, can’t: Summit Technology takes a first-layer approach to crawl health, rather than treating it as an afterthought, for the businesses it serves. If traffic has fallen for no apparent reason, traffic flow investigations should begin with the crawl layer.