Key takeaways
- A Screaming Frog audit should start from a saved configuration profile, not from default settings.
- Use Spider mode for the full crawl, List mode for URL sets such as redirects or sitemaps, and Compare mode to show change.
- Crawl JavaScript-heavy sites twice, text only and rendered, to see what depends on scripts.
- The client gets a prioritised issue list, not the raw export.
A Screaming Frog audit is the backbone of most technical reviews our team delivers through white label SEO. When you supervise a team of specialists working on 18 online stores with more than 3 million products, the only way to keep audits comparable is to make everyone crawl the same way. This guide is the crawl workflow, step by step. The full list of what to check across a site lives in our technical SEO audit checklist; here we focus on running the tool well.
How should you configure Screaming Frog for a client audit?
Build one agency configuration profile and save it, then adjust per client. Set the user agent, rendering mode, crawl speed, include and exclude patterns, and API connections to Search Console and Analytics. Save the profile so every specialist starts from the same settings, and note any client-specific change in the audit file.
The configuration guide covers each setting. The ones we always review: the user agent under HTTP Header (Googlebot presets exist), the rendering mode (text only or JavaScript), Max URI/s to limit the request rate on fragile servers, and include or exclude regex to keep the crawl inside scope. Excluded URLs are not crawled at all, so use exclude carefully. Robots.txt is obeyed by default, which is what you want for an audit of what Google can reach. Profiles are saved from Config, Profiles, Save As, as described in the general user guide, which also recommends database storage mode, where crawls are saved automatically as they run.
- Ask the client for the crawl window and any rate limit before starting.
- Whitelist the crawler's IP or user agent if a firewall or CDN blocks bots.
- Connect Search Console and Analytics so traffic and clicks sit next to each URL.
- Enable crawling of linked XML sitemaps to compare sitemap and crawl.
- Record the profile name and settings in the audit notes.
Which crawl mode should you use?
Use Spider mode for the main crawl from the home page. Use List mode when you have a fixed set of URLs, such as old URLs to test redirects or a sitemap to validate. Use Compare mode to show what changed between two crawls, for example before and after a release or a staging site against production.
| Mode | Use it for | Deliverable |
|---|---|---|
| Spider | Full site crawl from a start URL | Main audit data |
| List | Redirect maps, sitemap URLs, top landing pages | Status of a defined URL set |
| Compare | Before and after, staging vs live | Added, new, removed and missing URLs |
In the list mode tutorial, crawl depth is set to 0, so only the uploaded URLs are crawled, and you can load URLs from a file, a paste or an XML sitemap. For migrations, turn on always follow redirects to see full chains. The crawl comparison guide explains Compare mode: it needs database storage mode and a licence, groups changes as added, new, removed and missing, and can map URLs with regex so a staging hostname can be compared with the live site. Change detection adds checks for titles, word count, crawl depth and internal links.
How do you crawl JavaScript sites in Screaming Frog?
Run the crawl twice: once in text only mode and once with JavaScript rendering. Compare titles, links, canonicals and content between the two. Anything that exists only in the rendered crawl depends on Google rendering the page, which works but adds risk and delay. Flag critical elements that appear only after rendering.
JavaScript rendering executes scripts and inspects the rendered page for links and titles, according to the rendering settings. Rendered crawls are slower, so limit them to key templates on large sites. Google's JavaScript SEO basics explain why the difference matters; our JavaScript SEO guide covers the fixes.
What should you export from the crawl?
Export the Issues tab, then the specific reports behind your top findings: response codes, redirects, canonicals, directives, duplicate titles and sitemaps. Save Multi-Export selections as a preset so every audit produces the same files. Keep the saved crawl itself for comparison next month. Exports are evidence; the client does not need all of them.
The Issues tab classes each item as an issue, opportunity or warning with a high, medium or low priority, and Screaming Frog itself notes that a specialist should still judge what matters. The exporting guide covers bulk exports, an issues export with one sheet per issue, Multi-Export presets that also work with scheduling, and Google Sheets output, which truncates large tabs because Sheets has a cell limit. For very large sites, export to CSV instead. Scheduling, also in the general user guide, lets you rerun the same profile as a one-off or at chosen intervals. The same guide notes the free version crawls up to 500 URLs at a time; saved configuration, scheduling and crawl comparison need a licence.
What goes into the client deliverable?
A prioritised list of problems, each with the affected URL count, an example URL, why it matters, the fix and the owner. Attach the supporting exports, not the full crawl. Lead with the three to five issues with the biggest likely impact on traffic or conversions, and park cosmetic items at the end.
Priority list
Issue, URL count, example, impact, fix, owner and effort for each finding.
Evidence pack
Only the exports that support the findings, named the same way every month.
Crawl comparison
From month two, a Compare-mode summary of what changed since the last crawl.
Crawl data shows what a crawler can reach; it does not show what Googlebot actually does. Pair it with log file analysis on large sites. Our white label SEO audit follows this workflow under your brand, or you can book a 30-minute call.
Frequently asked questions
Is the free version enough for client audits?
The free version crawls up to 500 URLs and lacks features such as saved configuration, scheduling and crawl comparison, so most agencies need a licence. For agency work, the paid version saves time and gives more consistent results across clients.
How long does a crawl take?
It depends on site size, server speed, rate limits and rendering. Rendered crawls take much longer. Test a sample crawl first to estimate the time, and agree on crawl speed with the client for large or fragile sites. Schedule large crawls overnight or on weekends so the audit doesn't affect real users.
Should we crawl as Googlebot?
It helps reveal bot-specific behaviour, but some firewalls block fake Googlebot requests. Agree on the user agent with the client. Use the standard crawler user agent first, then a Googlebot test where the server allows it, and note any differences in the audit.
Can the crawl harm the client's server?
An aggressive crawl can strain small servers. Limit Max URI/s and crawl outside peak hours. Ask the client or host about server limits before starting, especially on shared hosting. If you see errors or slowdowns, pause the crawl and lower the speed before continuing.
How often should we recrawl?
Monthly for active clients, plus a crawl before and after every major release. Compare each crawl with the previous one to spot new errors, missing pages or changes in titles and canonicals. Scheduled crawls make this easy to repeat without manual setup.
Want these agency guides first? Choose White Label SEO LATAM as a preferred source on Google, and my articles are more likely to show up for you in Top Stories.
Follow on GoogleAdd as a preferred sourceSources
- Screaming Frog, SEO Spider user guide: General
- Screaming Frog, SEO Spider user guide: Configuration
- Screaming Frog, SEO Spider user guide: Tabs
- Screaming Frog, How to use list mode
- Screaming Frog, How to compare crawls
- Screaming Frog, Exporting (SEO Spider guide)
- Google Search Central, Understand JavaScript SEO basics