Key takeaways
- Log file analysis for SEO means reading the server's record of every Googlebot request.
- Verify Googlebot by reverse and forward DNS or Google's published IP ranges; the user agent alone is not proof.
- Start with Search Console's Crawl Stats report; use logs when you need every URL and every response.
- Logs matter most on large sites: stores, directories, sites with faceted navigation.
Log file analysis is a technical deliverable most small agencies never sell, and most clients with large sites need at least once. It belongs in the same toolkit as the technical SEO audit, and we deliver it under your brand as part of white label SEO. This guide covers what logs show and how to run the project with a client; the strategy for managing crawl on big sites is in faceted navigation and crawl budget.
What is log file analysis in SEO?
It is the review of a web server's access logs to see how search engine crawlers request a site. Each line records a request: the IP address, date and time, the URL, the HTTP status returned, the size and the user agent. Filtering those lines for verified Googlebot shows which pages Google crawls, how often and with what result.
Most access logs follow a common or combined format, and a CDN may keep its own logs separate from the origin server. Ask which one the client has before you start, because the CDN is often the only place that sees every request.
What can server logs show that Search Console cannot?
Every request, not a sample. Logs show exactly which URLs Googlebot hits, including parameter and filter URLs, redirects it keeps following, errors it meets, and pages it never visits. You can match crawl frequency against your list of important pages and see whether crawl is spent on the pages that earn money.
| Question | Crawl Stats report | Server logs |
|---|---|---|
| How much does Google crawl? | Totals and trends | Exact counts per URL and day |
| Which URLs are crawled? | Example URLs only | Every requested URL |
| Which errors does Google hit? | Grouped by response | Each URL with its status |
| Which pages are never crawled? | Not shown | Compare logs with the sitemap or a crawl |
| Which crawler type? | Smartphone, desktop, image and others | Full user agent, verifiable by IP |
Search Console's Crawl Stats report groups requests by response, file type, purpose (discovery or refresh) and Googlebot type, but its help page says example URLs are not comprehensive. It is only available for root-level properties, and Google says sites with fewer than a thousand pages should not need it. That is a useful threshold for deciding when logs are worth the effort too.
How do you verify that a request really came from Googlebot?
Do not trust the user agent, which anyone can copy. Run a reverse DNS lookup on the IP, check the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup and confirm it returns the same IP. For large logs, match IPs against the IP range JSON files Google publishes instead.
Both methods are in Google's guide to verifying Googlebot, which lists separate files for common crawlers such as Googlebot, special crawlers such as AdsBot and user-triggered fetchers. Google's overview of crawlers and fetchers explains that Google identifies its crawlers by user agent, source IP and reverse DNS hostname, and that its crawlers support HTTP/1.1 and HTTP/2. Filtering out fake Googlebot traffic first keeps the rest of the analysis honest.
What should you look for in log files on large sites?
Look for crawl spent on URLs that should not exist, such as filter and parameter combinations, redirect chains and old URLs. Then look for the opposite: important pages Googlebot rarely or never requests. Add server errors and slow responses, which can make Google crawl less. Compare all of it with the pages that bring revenue.
- Share of Googlebot requests on parameter, filter and search URLs.
- Status codes: 3xx chains, 404s on linked URLs, 5xx spikes by date.
- Important templates (categories, products, locations) and how often each is crawled.
- Pages in the sitemap with no Googlebot requests in the period.
- Crawl of robots.txt, sitemaps and JavaScript or CSS resources.
- Changes in crawl after a release, migration or traffic drop.
Google's crawl budget guide says it is intended mainly for sites with 1 million+ unique pages changing about weekly, or 10,000+ pages changing daily, and calls those numbers a rough estimate. Those are the clients where logs pay off most: large stores, directories and listing sites. We have supervised the SEO of 18 US online stores with 3M+ products and worked on a 70,000-page directory, which is the scale at which crawl questions stop being theoretical. If logs show a sudden change, pair them with the traffic drop diagnosis steps.
How do agencies get server logs from clients?
Ask the client's developer or host for raw access logs, ideally 30 days or more, for every server or CDN that serves the site. Agree the format, the delivery method and how long you will keep the files. Logs contain visitors' IP addresses, so treat them as personal data and remove or mask what you do not need.
Who to ask
The developer, the hosting company or whoever manages the CDN.
How much
At least a few weeks, so you see patterns rather than one day.
Privacy
Agree storage, access and deletion in writing; mask visitor IPs if you can.
Filter
Keep verified search engine crawler lines; drop the rest early.
Privacy rules differ by market, so check the client's privacy policy and local law before the files arrive, and write the handling terms into your agreement. For most SEO questions you only need the crawler lines, which makes it easy to delete everything else on day one.
How do you sell log file analysis to a client?
Sell it as a one-time diagnostic for large or complex sites, with a clear question to answer: why new pages are slow to be indexed, why crawl rose after a release, or where Google spends its time. Deliver a short report with three to five fixes ranked by impact, then repeat it after the fixes ship.
Our white label technical SEO team can run log analysis under your brand. To talk about a specific client, book a 30-minute call.
Frequently asked questions
Do small sites need log file analysis?
Rarely. Google says sites under a thousand pages should not need the Crawl Stats report, and the same logic applies to logs.
Which tools read log files?
Dedicated log analyzers, crawler tools with log import, or a spreadsheet or database for smaller files.
How far back should logs go?
At least several weeks; longer if you are comparing before and after a release.
Can logs show AI crawlers too?
Yes. Any bot that requests pages appears in the logs with its user agent and IP.
Is log analysis a monthly deliverable?
Usually not. It works best as a diagnostic, repeated after major changes.
About the author
Juan José Giraldo is a senior SEO specialist with 5+ years of experience. He works on technical and local SEO for 600+ US car dealerships and supervises the SEO of 18 US e-commerce stores. He runs White Label SEO LATAM from Medellín, Colombia, and delivers white label SEO for agencies in the US, UK, Canada and Australia. Read more · LinkedIn.