Who needs Screaming Frog when you can go to Agent.ai?
This agent finds all blog URLs on a given domain.
Input: the domain + blog path (optional)
Output: a copyable table containing all URLs
Notes:
- Adding a blog path (like
/blog/) helps speed up the script, but isn't needed. It works with or without backslashes. - If path isn't specified, it will return pages with the following paths - 'blog', 'article', 'post', 'blogs', 'articles', 'posts', 'resources', 'resource'
- The tool is specifically built for relatively-static sites, like company domains. It does not work for sites with a lot of dynamic or USG content, like social networks / news / etc.
- Some sites block bots using CDNs like Cloudflare. Not much we can do there.
- Make sure to add suffixes as needed (like blog.nyckel.com vs just nyckel.com).
Sign up to work with this agent
Get full access to run, customize, and share this agent — plus hundreds more.
or Log in