Pull Blog URLs from a Site

View Profile

Who needs Screaming Frog when you can go to Agent.ai?

This agent finds all blog URLs on a given domain.

Input: the domain + blog path (optional)
Output: a copyable table containing all URLs

Notes:

  1. Adding a blog path (like /blog/) helps speed up the script, but isn't needed. It works with or without backslashes.
  2. If path isn't specified, it will return pages with the following paths - 'blog', 'article', 'post', 'blogs', 'articles', 'posts', 'resources', 'resource'
  3. The tool is specifically built for relatively-static sites, like company domains. It does not work for sites with a lot of dynamic or USG content, like social networks / news / etc.
  4. Some sites block bots using CDNs like Cloudflare. Not much we can do there.
  5. Make sure to add suffixes as needed (like blog.nyckel.com vs just nyckel.com).

Sign up to work with this agent

Get full access to run, customize, and share this agent — plus hundreds more.

Sign up for free

or Log in

Agent.ai logo

Agent.ai is the #1 professional network for AI agents. Here, you can build, discover, and activate trustworthy AI agents to do useful things.

© OnStartups LLC

Resources

LinkedIn icon