A shop's sitemap is the list of pages it wants found, which makes it the cheapest way to discover a whole catalogue. Scrapewise reads the sitemap, follows the index files, filters down to product pages, and then scrapes them into one table.
The standard caps a sitemap at 50,000 URLs, so large shops publish an index that points at a dozen gzipped children. A reader that only parses the first file finds a fraction of the catalogue and reports it as the whole thing.
Blog posts, category pages, filter combinations and static pages sit in the same sitemap. Scraping all of them costs pages you did not want and fills the output with rows that have no price.
Delisted products stay listed for weeks, and new ones appear before the sitemap is regenerated. A sitemap is a good starting list, not an inventory, so the run has to tolerate URLs that no longer resolve.
Following links from the homepage means fetching every category and filter page to find the products behind them. The sitemap gives you the same destination list without paying for the pages in between.
One sitemap address is enough to discover a whole catalogue
Ready data APIs for marketplaces and search, plus AI scrapers for any other shop
Free requests on every new account, no card needed
Discovery and scraping are two separate steps on purpose: you see the link list, and decide what it costs, before anything is scraped.
Give the sitemap address, or just the domain and let Scrapewise find it from robots.txt. Sitemap index files are followed to their children, and gzipped sitemaps are unpacked.
Keep only the URLs that match a path pattern, so product pages stay and blog, category and filter URLs drop out. You see the count before the run, which is the count you will be charged for.
The filtered list becomes the run's link list. Name the fields you want back: price, sale price, currency, title, stock, pack size, EAN. Pages that fail are retried and are not charged.
Re-read the sitemap daily or weekly to pick up new products and spot ones that disappeared. Group rules clean every run the same way, so the columns stay comparable over time.
You bring one address and a path filter. You get a product URL list, then a row per URL.
A sitemap.xml, a sitemap index, or just the domain — robots.txt is checked for the sitemap address.
https://shop-a.example/sitemap.xmlA path pattern so products stay and blog or category pages drop out. Leave it empty to keep everything.
/product//p/-p-Name the columns for the scrape step. Anything the product page shows can be a column.
pricetitlestockeanHow often the sitemap is read again to pick up new and removed products.
One row per kept product URL, 8 columns each
CSV or Excel export, REST API, or MCP for AI agents.
Illustrative figures in the shape of a mid-size shop, not a customer's site. The harvest step produces the link list; the scrape step runs on the kept column.
| Sitemap file | URLs listed | Kept after filter | Why |
|---|---|---|---|
| sitemap.xml (index) | 11 children | — | Index file, followed to its children |
| sitemap_products_1.xml.gz | 50,000 | 48,310 | Product paths kept, 1,690 were redirects |
| sitemap_products_2.xml.gz | 12,744 | 12,201 | Product paths kept, 543 already delisted |
| sitemap_categories.xml | 4,120 | 0 | Category pages: no price to read |
| sitemap_filters.xml | 38,900 | 0 | Filter combinations: duplicate products |
| sitemap_blog.xml | 870 | 0 | Editorial pages |
| sitemap_brands.xml | 612 | 0 | Brand landing pages |
| sitemap_static.xml | 41 | 0 | About, delivery, terms |
| sitemap_images.xml | 96,004 | 0 | Image sitemap, not pages |
| Total | 203,291 | 60,511 | 60,511 product pages queued to scrape |
One sitemap address becomes every product URL on the site, with the index files followed and the gzip unpacked for you.
Harvest and scrape are separate steps, so the filtered URL count is visible first and that is the number of pages the run will cost.
Re-reading the sitemap on a schedule surfaces products that appeared and ones that stopped being listed, instead of a flat snapshot.
Read its sitemap. Give Scrapewise the sitemap address or just the domain, and it follows the sitemap index to every child file, unpacks gzip, and returns the URLs. A path filter keeps the product pages and drops blog and category URLs.
That is the normal case above 50,000 URLs, and it is handled. The top-level file is a sitemap index listing children, often gzipped. All of them are followed and merged into one link list, with the source file kept as a column.
A sitemap is published by the site for crawlers to read and its address is usually declared in robots.txt. Scrapewise reads public pages only, at a rate that does not interfere with the site, and does not touch anything behind a login.
Point Scrapewise at a sitemap, see the filtered product URL count, then decide what to scrape. Every new account gets 5 free requests, and after that you pay as you go from a prepaid balance. Failed pages are not charged.
How the sitemap is read, what gets filtered, and what it costs.
A sitemap scraper reads a site's sitemap.xml to discover the pages it publishes, then scrapes those pages for data. It is the cheapest way to cover a whole catalogue, because the site has already listed every product URL for crawlers to find.