[{"data":1,"prerenderedAt":236},["ShallowReactive",2],{"learn-lesson-competitor-price-monitoring-find-competitor-product-urls":3},{"course":4,"lesson":68,"index":192,"outline":193,"prev":234,"next":235},{"slug":5,"order":6,"level":7,"time":8,"card_text":9,"seo":10,"hero":16,"outcomes":26,"who":36,"syllabus":47,"faq":50,"lessonCount":67},"competitor-price-monitoring",1,"No coding required","8 lessons, about 90 minutes","From \"we check three competitors by hand on Mondays\" to a feed you trust enough to reprice from. The eight decisions in order, including the two that quietly ruin most projects.",{"title":11,"description":12,"keywords":13,"og_title":14,"og_description":15},"Competitor Price Monitoring: A Free 8-Lesson Course","Build a competitor price monitoring pipeline end to end: pick competitors, find product URLs, extract prices, match to your catalogue, schedule, export, reprice. Free, ungated.","competitor price monitoring, price monitoring course, how to track competitor prices, price tracking pipeline, competitor price tracking tutorial, retail price monitoring","A free course on building a competitor price monitoring pipeline","Eight written lessons on the part nobody covers: matching, data quality, and turning prices into decisions.",{"badge":17,"title":18,"subtitle":19,"cta_primary":20,"cta_secondary":23},"Course one","Build a competitor price monitoring pipeline","Price monitoring looks like a scraping problem for about a week. Then you discover that scraping was the easy part, and the project actually lives or dies on which competitors you picked, whether their listings are really the same product as yours, and whether anyone notices the morning the feed comes back half empty. This course is those eight decisions, in the order you have to make them.",{"label":21,"url":22},"Start with lesson one","/learn/competitor-price-monitoring/what-is-competitor-price-monitoring",{"label":24,"url":25},"Try the free price checker","/tools/free-competitor-price-checker",{"title":27,"items":28},"What you will be able to do",[29,30,31,32,33,34,35],"Say which competitors matter to your margin, and defend the list to a finance director","Collect competitor product URLs without copying them one at a time","Decide which fields you actually need, and which ones are a trap","Judge a match rate honestly, including the denominator that most vendors quietly change","Spot a feed that has silently degraded before a buyer acts on it","Put the numbers where the decision is made, instead of in a dashboard nobody opens","Write a repricing rule that survives contact with a competitor who is out of stock",{"title":37,"for_title":38,"for":39,"not_title":43,"not_for":44},"Who this is for","Written for",[40,41,42],"Pricing, category and e-commerce managers who own the number","Marketplace sellers repricing against a handful of named rivals","Anyone evaluating price monitoring vendors who wants to ask better questions","Not written for",[45,46],"Engineers who want a tutorial on CSS selectors and headless browsers — that is a different and much shorter problem","Anyone looking for a sub-second price feed; nothing here runs faster than daily",{"title":48,"intro":49},"The eight lessons","Each one ends where the next one starts. Four lessons are pure method; the ones that walk through our product say so in their own heading.",{"badge":51,"title":52,"description":53,"items":54},"FAQ","Before you start","Questions people ask in the first ten minutes.",[55,58,61,64],{"title":56,"description":57},"Do I need a Scrapewise account to follow this?","For five of the eight lessons, no. Lessons three, four and seven walk through doing the thing in our portal and are marked as needing an account. A new account comes with five free requests and no card, which covers those lessons.",{"title":59,"description":60},"Is scraping competitor prices legal?","This is not legal advice. Reading a publicly visible price is not the same question as storing it, republishing it, or building a product on it, and the answer changes by jurisdiction and by the site's own terms. Lesson two covers how to keep the collection narrow enough that the question stays simple, but take your own advice before you start.",{"title":62,"description":63},"How often can prices be checked?","This course assumes a daily cycle, which is what most retail pricing decisions actually run on. If you need minute-level reaction — a marketplace buy box, an airline fare — the architecture in lesson six is wrong for you and you want an event-driven system instead.",{"title":65,"description":66},"What if my competitors are not online retailers?","The method holds for any public price list: distributors, wholesalers, rental catalogues. What changes is lesson three, because those sites rarely have a clean product sitemap. Lesson three covers that case directly.",8,{"slug":69,"nav_title":70,"title":71,"summary":72,"time":73,"needs_account":74,"seo":75,"blocks":79,"takeaways":183,"next_step":188},"find-competitor-product-urls","Finding product URLs","Finding every competitor product URL without copying them by hand","Four ways to get a competitor's full product URL list, ranked by how much work they are, and what to do when none of them work.","12 min",true,{"title":76,"description":77,"keywords":78},"How to Get a Competitor's Full Product URL List","Four methods for collecting every product URL on a competitor site — sitemaps, category crawling, search, and a URL extractor — plus what to do when the site hides them.","find competitor product urls, extract product urls, product url list, scrape product links, xml sitemap products, competitor catalogue",[80,85,101,109,116,145,149,168,178],{"type":81,"paragraphs":82},"prose",[83,84],"A scraper needs addresses. Before anything can read a price, something has to produce a list of competitor product pages — ideally the complete list, ideally kept up to date as they add and drop lines.","This is the step teams most often do by hand, and it is the step where doing it by hand hurts most: a thousand URLs copied from a browser is a week of somebody's life, and it is stale by the time it is finished. There are four ways to avoid that, and they are worth trying in this order.",{"type":86,"title":87,"items":88},"steps","The four methods, easiest first",[89,92,95,98],{"title":90,"text":91},"1. The XML sitemap","Most e-commerce platforms publish one, because Google needs it. Try /sitemap.xml and /robots.txt on the competitor's domain; robots.txt usually names the sitemap even when the default path is wrong. What you get is often a sitemap index pointing at a dozen child files, one of which is products. This is the best possible outcome: a complete, maintained, machine-readable list of every product page, published deliberately. On a well-run store it takes about five minutes.",{"title":93,"text":94},"2. Category pages","When there is no product sitemap, walk the category tree instead: open a listing page, take every product link on it, follow the pagination, repeat. Slower and noisier than a sitemap — you will pick up promotional tiles and cross-sell blocks — but it works on nearly every store, and it has one genuine advantage: it tells you which category the competitor files a product under, which is useful context for matching later.",{"title":96,"text":97},"3. Their own site search","If you only care about a few hundred known products, searching the competitor's site for each EAN or manufacturer part number is often the fastest route, and it gives you something the other methods do not: a direct, high-confidence link between their URL and your SKU. That link is worth a great deal in lesson five. The limitation is obvious — it only finds what you already know to look for.",{"title":99,"text":100},"4. A URL extractor","A tool that takes the domain and returns the product URLs, handling the sitemap-or-crawl decision for you. We publish a free one; so do others. This is method one and two with the plumbing hidden, which is worth it when you are doing this for eight competitors rather than one.",{"type":102,"variant":103,"title":104,"text":105,"cta":106},"callout","product","Doing it with our free tool","The Competitor Product URL Extractor takes a store URL and returns the product page addresses it can find, with no account and no card. It is the fastest way to find out whether a given competitor is going to be easy or hard before you commit to tracking them. Note the honest limitation: stores that publish no sitemap and render their listings entirely in JavaScript can come back with nothing.",{"label":107,"url":108},"Open the free URL extractor","/tools/free-product-url-extractor",{"type":81,"title":110,"paragraphs":111},"Cleaning the list",[112,113,114,115],"Whatever method produced it, the raw list is never the list you want. Three passes fix most of it.","First, strip tracking parameters. A URL ending in ?utm_source=newsletter is the same page as the one without it, but a naive pipeline will treat them as two pages and bill you for both. Cut everything after the question mark unless you know a parameter is load-bearing — on some stores the variant selector lives there, and you will see it because the page genuinely changes when you remove it.","Second, collapse variants. A t-shirt in six sizes is often six URLs with the same price. Decide deliberately whether you are tracking the product or the variant, because tracking all six multiplies your bill by six and usually tells you one thing. Where sizes really are priced differently — shoes, tyres, anything sold by capacity — you do want them separately, and you should say so explicitly rather than letting the extractor decide.","Third, drop what is not a product. Category pages, blog posts, gift cards, bundles, store locators. They sneak in from every method and each one is a page you pay to read and a row that will never match.",{"type":117,"title":118,"intro":119,"headers":120,"rows":124},"table","Choosing a method per competitor","In practice you will use different methods for different sites, and that is fine.",[121,122,123],"Situation","Use","Watch out for",[125,129,133,137,141],[126,127,128],"Product sitemap exists and is current","Sitemap","Sitemaps that include discontinued lines for months",[130,131,132],"No sitemap, normal server-rendered listings","Category crawl","Infinite scroll with no page numbers",[134,135,136],"You only track 200 known EANs","Their site search","Zero-result pages returning a 200 status",[138,139,140],"Eight competitors, limited patience","URL extractor","JavaScript-only storefronts returning nothing",[142,143,144],"Login required to see prices","None of these","Do not. A price behind a login is not a public price.",{"type":102,"variant":146,"title":147,"text":148},"warning","Stop at the login wall","If a competitor's prices are only visible to registered trade customers, the whole exercise changes character — from reading a public page to accessing an account under terms somebody agreed to. This course does not cover that, and you should talk to whoever handles your legal questions before anyone on your team creates that account.",{"type":86,"title":150,"intro":151,"items":152},"Worked example: one competitor's sitemap to a payable URL list","Start to finish on a single competitor, with the figure each step removes. The absolute counts will differ for your sites; the shape of the reduction will not.",[153,156,159,162,165],{"title":154,"text":155},"Fetch /sitemap.xml and follow the index","Most sitemaps are an index pointing at several child files. You want only the product ones — if they are named products-1.xml through products-4.xml, take all four and none of the blog, category or store-locator files. Say the four together list 41,000 URLs.",{"title":157,"text":158},"Strip query strings and fragments, then deduplicate","Remove everything after ? and #. Tracking parameters, sort orders and session ids turn one page into several, and every duplicate is a page you pay to fetch twice. Say this leaves 38,600 distinct URLs.",{"title":160,"text":161},"Keep only the paths that match the product pattern","Open three product pages by hand and find the shared segment — /p/, /product/, /dp/, or a trailing numeric id. Everything that does not match is a category, a filter permutation or an article. Say 29,000 survive.",{"title":163,"text":164},"Decide products or variants, and write the decision down","If the sitemap lists every colour and size separately, those 29,000 URLs may be 9,000 products. Collapsing to the parent cuts the fetch count by roughly two-thirds and loses per-variant stock. Collapsing is usually right for pricing and usually wrong for availability, which is why the decision belongs in writing rather than in whoever built it.",{"title":166,"text":167},"Intersect with the SKUs you already chose","You picked 1,200 SKUs in lesson two. Only the overlap is worth fetching. If 700 of your 1,200 appear in their list, this competitor costs 700 pages a run, not 29,000 — and that single intersection is the difference between a project and a quote you walk away from.",{"type":169,"title":170,"intro":171,"items":172},"list","What usually goes wrong","Discovery is the cheapest stage to get right and the most expensive to get wrong, because the error multiplies by every future run.",[173,174,175,176,177],"Taking the whole sitemap. It is the single most expensive mistake in this lesson: 29,000 pages a run instead of 700, every run, forever.","Missing the sitemap index and reading only the first child file, then wondering why a third of the catalogue never appears.","Leaving tracking parameters attached. The same product arrives three times under three URLs, and the deduplication problem moves downstream into matching, where it is far harder to see.","Discovering URLs once and never again. Competitors add and drop lines constantly, so a list built in January is measurably wrong by March and silently wrong the entire time.","Pushing past a login wall because the data behind it is better. That is the one method on this list with a legal answer rather than a technical one, and course six is where it gets answered.",{"type":81,"title":179,"paragraphs":180},"Keeping the list fresh",[181,182],"Catalogues move. A competitor adds forty lines before Christmas and drops a hundred in January, and a URL list captured once decays at maybe one to three percent a month depending on the category. The symptom is subtle: your page count stays flat while the number of pages that return a real price slowly falls, and your coverage quietly erodes.","The fix is boring and effective: re-run whichever method you used on a schedule — monthly is usually enough — and diff the result against the current list. New URLs get added, URLs that have disappeared from the sitemap for two consecutive runs get retired. That diff is also a small, genuinely interesting competitive signal in its own right: it is a list of what your rival started and stopped selling.",[184,185,186,187],"Try the XML sitemap first. It is complete, maintained and takes five minutes.","Strip tracking parameters and decide explicitly whether you track products or variants — both affect the bill directly.","Different competitors warrant different methods; there is no need to pick one for all of them.","Re-run the discovery monthly and diff it. The diff doubles as a record of what your competitor launched and dropped.",{"text":189,"label":190,"url":191},"With addresses in hand, the next question is what to read off each page — and which tempting fields are a trap.","Lesson 4: extracting price, stock and shipping","/learn/competitor-price-monitoring/extract-price-stock-and-shipping",2,[194,201,207,208,213,219,224,229],{"slug":195,"navTitle":196,"title":197,"summary":198,"time":199,"needsAccount":200},"what-is-competitor-price-monitoring","What it actually is","What competitor price monitoring actually is","The four stages of a price pipeline, why only two of them are scraping, and the one question to ask before you build anything.","9 min",false,{"slug":202,"navTitle":203,"title":204,"summary":205,"time":206,"needsAccount":200},"choose-competitors-and-skus","Choosing what to track","Choosing which competitors and which SKUs to track","How to build a list that is small enough to afford and large enough to matter, using margin at risk rather than gut feel.","11 min",{"slug":69,"navTitle":70,"title":71,"summary":72,"time":73,"needsAccount":74},{"slug":209,"navTitle":210,"title":211,"summary":212,"time":73,"needsAccount":74},"extract-price-stock-and-shipping","Extracting the fields","Getting price, stock and shipping off the page","Which fields to extract, why the sale price is two fields and not one, and the four ways a price appears on a page.",{"slug":214,"navTitle":215,"title":216,"summary":217,"time":218,"needsAccount":200},"match-listings-to-your-catalogue","Matching to your catalogue","Matching competitor listings to your own catalogue","The stage that decides whether your feed is intelligence or fiction, and the denominator trick that makes bad match rates look good.","13 min",{"slug":220,"navTitle":221,"title":222,"summary":223,"time":206,"needsAccount":200},"schedule-runs-and-catch-silent-failure","Scheduling and data quality","Scheduling runs and catching silent data loss","How often to actually check, and the four alerts that catch a degrading feed before someone reprices from it.",{"slug":225,"navTitle":226,"title":227,"summary":228,"time":199,"needsAccount":74},"export-to-sheets-bi-and-erp","Getting the data out","Getting the data into Sheets, BI or your ERP","Four delivery routes ranked by how likely they are to actually get used, and the column contract that stops downstream jobs breaking.",{"slug":230,"navTitle":231,"title":232,"summary":233,"time":73,"needsAccount":200},"turn-price-data-into-repricing-rules","From data to decisions","Turning price data into repricing decisions","Why \"match the cheapest\" destroys margin, what a rule needs besides a competitor price, and how to start without automating anything.",{"slug":202,"navTitle":203,"title":204,"summary":205,"time":206,"needsAccount":200},{"slug":209,"navTitle":210,"title":211,"summary":212,"time":73,"needsAccount":74},1791047866754]