Live catalogue scan Free · no sign-up

Extract every product URL from a competitor's category page

Paste one category or collection URL. We read the page live, work out which links are product pages, and give you the product URLs plus the URL structure behind the catalogue.

Point it at a listing page, not the homepage — that's where the products are. Live fetch — nothing cached

What comes back

Product pages

Every link on the page we judged to be a product, listed with the title we saw and why we called it one.

/{…}

URL templates

The patterns behind the catalogue — how many products sit under each shape of URL.

Published data fields

Which fields the site exposes per product in its structured data: price, SKU, GTIN, brand, stock.

Example shape only — every number above is produced from the page you paste.

Up to 10 pages

Read per scan, never the whole site

One category

The products you asked for, not the sitemap

Free

No sign-up, no card, no email

HOW IT WORKS

How the product URL extractor works

A sitemap gives you every URL on a domain. That's the wrong answer to "what's in this category?" Here's what happens on every scan above.

Step 1

We read the one page you point at

A category, collection, or search-results page — the page that already lists the products you care about. One plain fetch of the real page, and nothing else on the domain is touched. If the site refuses us, we retry once by a different route.

1 URL in • one retry if blocked
Step 2

We work out what each link is

Every link is judged on its URL pattern, the markup the cart wrote around it, and any product structured data the page publishes — then labelled product, category, pagination, or other.

URL pattern + structured data
Step 3

You get the catalogue shape

Duplicates merged, every product page listed with the title we saw, the data fields the site publishes per product, and the URL templates behind the whole category.

CSV export
WORKED EXAMPLE

What a scan actually returns

One real run, printed exactly as it came back. This is allbirds.com/collections/mens-shoes on 21 September 2026 — a single category URL in, the whole catalogue shape out.

36 product-looking pages found

on allbirds.com · 1 of 10 pages read · 104 duplicate links merged · scanned in 1.70s

Retrieved directly
36 products26 categories13 other pages15 external

194 links were read on that page. 104 of them were the same products linked twice — once from the image, once from the title — which is why the count is 36 and not 140. Merging those duplicates is most of the work a spreadsheet makes you do by hand.

URL structure discovered

  • /products/{slug}product36
  • /collections/{slug}category23
  • /pages/{slug}other5

This is the part a sitemap cannot give you. One pattern holds every product in the category, so a monitoring job can be pointed at the pattern instead of at 36 individual URLs that change every season.

Data fields this site publishes

Read from structured product data on 34 of the 36 pages above. Two products publish no record at all — reported as 34, not rounded up to 36.

  • Brand34
  • Price34
  • Product name34
  • Product type34
  • SKU34
  • Variant / size34

Product pages

PageDetected asWhy
/products/mens-allbirds-flip-flopMen's Allbirds Flip Flop Mushroom $50Productproduct URL pattern · product link text · price printed next to this link
/products/mens-allbirds-slide-anthraciteMen's Allbirds Slide Anthracite $55Productproduct URL pattern · product link text · price printed next to this link
/products/mens-cruiser-dark-navyMen's Cruiser Dark Navy $105Productproduct URL pattern · product link text · price printed next to this link
/products/womens-cruiser-canvas-portCanvas CruiserProductproduct URL pattern · product link text

Four of the 36, with the evidence column as it came back. The last row is in here on purpose: 34 products had a price printed beside the link and two did not, and those two are still called products on the signals that did fire. The "why" column is what lets you see which signal carried a decision, instead of being handed a number to trust. The full list exports to CSV.

LIMITATIONS

When a category page can't be read — and why

This tool does one plain fetch per page, on purpose, so it can stay free. That choice has consequences, and they are predictable. Here is every way a scan comes back short, what causes it, and what it would take to get the products anyway.

The grid is drawn in the browser

Some storefronts ship an empty shell and build the product grid client-side from an internal API. A plain fetch sees the shell. The result comes back with almost no links at all — not "no products", but "no HTML worth reading" — and the page says so rather than reporting zero as an answer.

What it takes: Rendering the page in a real browser before reading it, which is what the paid engine does and what keeps this tool free.

Product URLs carry no recognisable pattern

Plenty of European retailers put products straight at the root: thomann.de/gb/studio_headphones.html returned 191 links and 0 products, because every one of them looks like /akg_k271_mkii.htm. There is nothing in the URL for a pattern-matching tool to key on, and the page reports that honestly instead of guessing.

What it takes: Classifying on page structure — the markup the cart wrote, the data layer, the price beside the link — rather than on the URL alone.

The site sends you to another country

A .de URL can land on .co.uk. That is a different assortment at different prices, so answering the second question as though it were the first would be wrong. The scan names the host it actually ended on instead of folding it into the site you typed.

What it takes: Nothing — but read the result as being about the country you were redirected to, not the one you asked for.

A bot wall answers instead of the page

Cloudflare, Akamai, DataDome and PerimeterX will serve a challenge to a plain request. The tool retries once through our proxy network and tells you it did. Proxied runs stop at 3 pages instead of 10, because each one costs real money and this tool has no sign-up to bill against.

What it takes: A proxy budget that scales with the job, plus browser rendering for the walls that challenge every request.

The listing page publishes no structured data

A category page can list 60 products and publish zero machine-readable records about them — or publish records that belong to a recommendations carousel rather than to the grid. Either way one fetch cannot tell you which fields this site exposes per product, and the tool distinguishes the two cases rather than asserting the stronger one.

What it takes: Fetching the product pages themselves, where the price, SKU and stock actually live.

The category is bigger than one scan

A run reads the page you gave it plus up to nine pages of its pagination, inside a 60-second budget. Deep catalogues hit one ceiling or the other. When that happens the result says it stopped early, because a short list that looks complete makes every number downstream of it quietly wrong.

What it takes: A scheduled job with no wall-clock budget, which is the difference between a free demo and monitoring.

Pasting a handful of category URLs by hand hits these walls occasionally. Watching a few thousand product pages a week hits them constantly, which is the problem automated competitor price tracking exists to solve. If a bot wall is what stopped you, we wrote up how the major ones behave in bypassing Cloudflare, Akamai and PerimeterX. And once you have the URLs, the free competitor price checker will compare any two of them side by side.

FAQ

Frequently asked questions

How the competitor product finder works.

How do I get all the product URLs from a website for free?

Paste the category page that lists them and press Find products. The tool reads that page live, decides which of its links are product pages, merges the duplicates that storefronts create by linking each tile twice, and gives you the list plus a CSV export. No sign-up, no card, no email — the limits are on what one scan reads, not on who is allowed to run it.

Why not just use the site's sitemap.xml?

Is it legal to extract a competitor’s product URLs?

Does it work on Shopify, WooCommerce and Magento stores?

Does it work on European storefronts, or only English-language sites?

Can I export the product URLs to CSV or Excel?

What should I paste in?

Why does it say "product-looking" instead of "products"?

Why did it only find a handful of products?

Does it crawl the whole website?

Do you check the prices?

AUTOMATE IT

Finding the pages is the easy part.

ScrapeWise watches every product you found and tells you the moment a price or stock level changes.