Catalog all the URLs on your site that you may want to optimize for users and search engines.
Crawling may take a few seconds or a couple of hours, depending on the size of your site. If your site has more than 20,000 pages, Screaming Frog may crash. Either save your progress periodically or consider a different tool.
If the crawler gets stuck in a subdirectory (/wp-content/) with thousands of irrelevant pages, pause and clear the crawl, add the subdirectory to the Exclude filter, and start the crawl again. You may need to repeat this step.
Alternatively, keep H1 and Word Count if you’re interested in reviewing those elements.
- Pages with UTM parameters
- Tag pages
- Author pages
- Pagination URLs.
Your content inventory should include only pages that marketers will actively try to improve.