Server logs reveal hidden crawl waste on large sites

Server logs are where the story gets concrete. They show the exact requests bots make to a site, which means agencies can stop guessing about crawl behavior and start proving where search engines are wasting time, skipping important pages, or running into infrastructure problems.

Why server logs matter on big sites

For large properties, the gap between what tools report and what bots actually do can be expensive. Google Search Central says crawl-budget management is most relevant for very large sites with 1 million-plus unique pages, medium-or-larger sites with 10,000-plus rapidly changing pages, and sites with many URLs labeled “Discovered - currently not indexed.” That is the kind of environment where a dashboard-only view breaks down fast.

Google also says crawl budget is shaped by crawl capacity limit and crawl demand. If a site slows down or starts returning server errors, Google’s crawl capacity can drop. For agencies, that turns server health into an SEO issue, not just a hosting issue, because wasted bot visits and reduced crawl capacity can both suppress discovery and recrawling.

The clients that justify log-file work

Log analysis is not for every brochure site. It becomes a strong fit when the site has enough scale, enough change, or enough URL clutter that crawl decisions matter financially. Ecommerce catalogs, large publishers, marketplaces, and enterprise sites with faceted navigation are the clearest candidates.

• Large ecommerce sites often generate endless low-value parameter URLs.

• Publishers can bury fresh content under old archive and taxonomy paths.

• Sites with frequent launches, promotions, or inventory churn need recrawling to keep up.

• Any property already showing lots of “Discovered - currently not indexed” pages has a strong reason to look deeper.

That is where agencies can frame log work as a premium diagnostic. It is not just a technical audit add-on; it is a way to prioritize fixes that influence how search engines spend attention across millions of URLs.

What logs reveal first

Server logs are valuable because they capture what happened, not what a crawler simulator inferred. Search Engine Land’s log-file guide notes that logs record the exact URL, timestamp, response status, user-agent, and IP address for each hit. That gives agencies a direct record of bot behavior across both historical and live activity.

The first problems that usually surface are the ones that eat crawl budget without improving indexation:

• Repeated hits to filtered navigation URLs.

• Redirect chains that force bots through extra hops.

• Slow responses that drag down crawl capacity.

• Unexpected 200 responses on URLs that should be gone.

• Crawlers spending time in low-value sections while important pages are not recrawled often enough.

On ecommerce sites, that can look like Googlebot revisiting the same faceted combinations over and over while product detail pages wait too long for another crawl. On publisher sites, logs can show outdated archive paths getting more attention than the latest stories, which is a clue that internal linking and information architecture are steering bots the wrong way.

How Googlebot actually behaves

Google’s own guidance helps agencies interpret those patterns correctly. Google says Googlebot usually should not access a site more than once every few seconds on average, which makes repeated rapid hits to the same area a sign that something in the structure is attracting more bot attention than it deserves. Google also says most Google Search crawl requests are made with Googlebot Smartphone, with a minority from Googlebot Desktop.

That matters operationally because the log file should not be treated as a generic bot dump. If mobile Googlebot is dominating crawl activity, agencies need to ask whether the site’s mobile experience, templating, and internal links are causing crawl imbalances. Google also says Googlebot discovers new URLs primarily from links embedded in previously crawled pages, which makes internal linking a core crawl-control lever rather than a cosmetic SEO task.

There is another detail agencies should remember when reviewing logs and oversized assets: Google says Googlebot crawls the first 2MB of supported file types and the first 64MB of PDFs. If logs show bot activity around oversized documents or heavy files, the crawl may be limited in ways the content team did not expect.

Bing brings a different control layer

Bing Webmaster Tools adds an operational wrinkle that agencies can use to their advantage. Microsoft says site owners can slow Bingbot down or speed it up by hour, which is especially useful when traffic spikes or site performance changes across the day. Microsoft also says a robots.txt crawl-delay directive overrides Bing’s crawl-control settings.

That gives agencies a practical conversation with clients about bot traffic management. If a site is under strain during peak business hours, crawl control becomes part of performance hygiene. If the site is quiet overnight, agencies can use that window to let Bingbot work harder without competing with customers.

Where logs fit alongside Semrush, Ahrefs, and crawler audits

Most agencies already rely on crawl platforms, including tools such as Semrush and Ahrefs, plus Google Search Console and Bing Webmaster Tools. Those tools are useful, but they are still partial views. Search Engine Land’s broader guidance draws the line clearly: crawl tools simulate crawler navigation, while logs reveal what bots actually did.

That difference is where the strategic value lives. A crawler can flag broken links, thin pages, or redirect loops in a model of the site. A log file can prove which URLs search engines actually requested, how often they returned errors, and which bot identities were involved, including newer names such as GPTBot and Applebot alongside Googlebot and Bingbot. For agencies trying to understand both traditional search crawling and the broader AI-crawler environment, that visibility is becoming a real differentiator.

How agencies can package log analysis as a higher-value service

The strongest pitch is not “we can read logs.” It is “we can find the crawl waste that standard platforms miss, then turn that into measurable technical priorities.” That changes the sales conversation from a one-time audit to an ongoing optimization service.

A practical agency package can look like this:

1. Establish baseline crawl behavior for the highest-value site sections.

2. Identify crawl waste, including parameters, redirects, and duplicate paths.

3. Map which bot traffic is supporting discovery and which is draining resources.

4. Tie findings to fixes the client can act on, such as internal linking, redirect cleanup, parameter handling, and server performance improvements.

5. Recheck logs after changes to prove whether crawl distribution improved.

That kind of reporting is easier to defend in a retention meeting than rankings alone. When a client has a site large enough to justify crawl-budget management, log analysis gives the agency a sharper way to show impact, explain technical risk, and protect the visibility gains that dashboard-only reporting can miss.