[{"data":1,"prerenderedAt":82},["ShallowReactive",2],{"$f41sATi_XYmzQYZTvO8d_LKq3QCt1xkO-PIi8iWelzG8":3},{"title":4,"date":5,"dateModified":6,"datePublished":7,"dateModifiedISO":7,"image":8,"content":9,"faq":10,"metaTitle":30,"metaDescription":31,"author":32,"authorBio":33,"authorLinkedin":33,"authorTitle":33,"authorPhoto":33,"lastReviewed":33,"researchBasis":33,"category":34,"readingTime":35,"related":36,"prev":52,"next":55,"toc":58,"takeaways":81},"How to Find the Hidden JSON API Behind a Shop Page (and Why It Is Cheaper)","30 Sep 2026","30 SEP 2026","2026-09-30","/img/news/find-hidden-json-api-shop-page-2026.png","\u003Cp>On most modern shops the price you see was not in the HTML the server sent. The page asked a JSON service for it, then drew it. If you can call that JSON service yourself, you get the price, the product id and often the EAN and stock, without parsing a single CSS class.\u003C/p>\n\u003Cp>We now look for this JSON first on every site we build a scraper for, and use HTML selectors last. This post is how to find the hidden API behind a shop page, which platform endpoints to try, and the tests we run before we trust one.\u003C/p>\n\u003Ch2 id=\"why-bother\">Why Bother\u003C/h2>\n\u003Cp>Three reasons, from our own builds.\u003C/p>\n\u003Cp>\u003Cstrong>It survives redesigns.\u003C/strong> A theme change moves the price into a different box. The JSON the theme reads from stays the same, because every theme on that platform reads it.\u003C/p>\n\u003Cp>\u003Cstrong>It is cheaper.\u003C/strong> On a recent yarn project, links read from Shopify's JSON cost about a ninth of the same links read with AI extraction. On a multi-site project, moving to listing APIs at their largest page and batch sizes was a big part of cutting daily requests by 88%. That story is in our \u003Ca href=\"/blogs/competitor-price-monitoring-case-study-2026\">competitor price monitoring case study\u003C/a>.\u003C/p>\n\u003Cp>\u003Cstrong>It is exact.\u003C/strong> A JSON field called \u003Ccode>price\u003C/code> next to a field called \u003Ccode>compare_at_price\u003C/code> is a lot less ambiguous than two numbers on a page, one of them crossed out.\u003C/p>\n\u003Caside class=\"article__usecase-card\">\u003Cdiv class=\"article__usecase-label\">Related use case\u003C/div>\u003Ch3 class=\"article__usecase-title\">Any-site data scraper\u003C/h3>\u003Cp class=\"article__usecase-blurb\">No-code extraction from any website. Managed infrastructure, no anti-bot headaches.\u003C/p>\u003Ca class=\"article__usecase-link\" href=\"/use-cases/data-scraper\">See how it works →\u003C/a>\u003C/aside>\u003Ch2 id=\"step-1-watch-what-the-page-asks-for\">Step 1: Watch What the Page Asks For\u003C/h2>\n\u003Cp>Open a category page or a product page in Chrome, open DevTools, go to the Network panel, and reload. Filter by Fetch/XHR, then search the requests for a price you can see on the page, for example \u003Ccode>24.95\u003C/code> or \u003Ccode>2495\u003C/code>. Chrome's docs cover the panel in \u003Ca href=\"https://developer.chrome.com/docs/devtools/network\">inspect network activity\u003C/a>.\u003C/p>\n\u003Cp>What you are looking for is a response with a list of products, each with a price and an id. Common shapes:\u003C/p>\n\u003Cul>\n\u003Cli>A search or merchandising service, often GraphQL, that returns the product grid with prices.\u003C/li>\n\u003Cli>A search API where each product carries its variants as child documents, so sizes, SKUs and EANs sit together.\u003C/li>\n\u003Cli>A batch API such as \u003Ccode>/api/products?ids=...\u003C/code> that takes a list of ids and returns details for each.\u003C/li>\n\u003Cli>A category page with an \u003Ccode>application/ld+json\u003C/code> block holding an \u003Ccode>ItemList\u003C/code> of products.\u003C/li>\n\u003C/ul>\n\u003Cp>Right-click the request and copy it as cURL. That command is your scraper. In Scrapewise you can paste it into the builder, or an agent can pass it to \u003Ccode>scrapewise_preview_scraper_from_curl\u003C/code> and get a data preview back.\u003C/p>\n\u003Ch2 id=\"step-2-try-the-platform-endpoints\">Step 2: Try the Platform Endpoints\u003C/h2>\n\u003Cp>If the network tab shows nothing useful, check what the shop runs on. The big platforms have public storefront JSON that every theme uses.\u003C/p>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Platform\u003C/th>\n\u003Cth>Endpoint to try\u003C/th>\n\u003Cth>What you get\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>Shopify\u003C/td>\n\u003Ctd>\u003Ccode>/products/<handle>.js\u003C/code>, or \u003Ccode>/variants/<id>.js\u003C/code> for one variant\u003C/td>\n\u003Ctd>Price, before-price, weight in grams, SKU, barcode, stock. Prices are in cents in \u003Ccode>.js\u003C/code>\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>WooCommerce\u003C/td>\n\u003Ctd>\u003Ccode>/wp-json/wc/store/v1/products?slug=<slug>\u003C/code>\u003C/td>\n\u003Ctd>Price in minor units with the number of decimals, stock, on-sale flag\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Magento\u003C/td>\n\u003Ctd>Open REST routes vary by shop. On one shop only \u003Ccode>/rest/V1/products-render-info\u003C/code> was open\u003C/td>\n\u003Ctd>Final price and regular price per product\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Any site\u003C/td>\n\u003Ctd>\u003Ccode>application/ld+json\u003C/code> in the page\u003C/td>\n\u003Ctd>Product name, price, currency, often GTIN and availability\u003C/td>\n\u003C/tr>\n\u003C/tbody>\u003C/table>\n\u003Cp>Shopify's product JSON is documented in the \u003Ca href=\"https://shopify.dev/docs/api/ajax/reference/product\">Ajax Product API\u003C/a>, and WooCommerce documents its \u003Ca href=\"https://developer.woocommerce.com/docs/apis/store-api/\">Store API\u003C/a>. Neither needs a key for public product data.\u003C/p>\n\u003Cp>Two Shopify details cost us time. The \u003Ccode>.json\u003C/code> variant endpoint has the weight in grams but no stock flag. The \u003Ccode>.js\u003C/code> endpoint has the stock state as well, but prices in cents. We use \u003Ccode>.js\u003C/code> and divide by 100 with an after-scrape rule. And a product's first variant is not always the one the customer means, so we fetch the exact variant from the link, not "the first one".\u003C/p>\n\u003Cp>On a competitor link list for a yarn retailer, 57% of the links ended up on Shopify's variant JSON and 2% on the WooCommerce Store API. The rest used microdata, JSON-LD, or AI extraction where nothing structured had the weight. The full build is in \u003Ca href=\"/blogs/price-per-unit-comparison-pack-sizes-2026\">comparing prices across pack sizes\u003C/a>.\u003C/p>\n\u003Caside class=\"article__inline-cta\">\u003Cp class=\"article__inline-cta-text\">Try ScrapeWise on your own URL. \u003Cstrong>Your first 5 requests are free.\u003C/strong>\u003C/p>\u003Ca class=\"article__inline-cta-btn\" href=\"https://portal.scrapewise.ai/login\" target=\"_blank\" rel=\"noopener\">Start Free →\u003C/a>\u003C/aside>\u003Ch2 id=\"step-3-don39t-forget-structured-data-in-the-html\">Step 3: Don't Forget Structured Data in the HTML\u003C/h2>\n\u003Cp>When there is no API, the page often still has a structured layer: JSON-LD, or \u003Ccode>itemprop\u003C/code> microdata on the price and currency. It is meant for search engines (Google explains it in its \u003Ca href=\"https://developers.google.com/search/docs/appearance/structured-data/product\">Product structured data\u003C/a> guide), so shops keep it correct, and it rarely changes with the design.\u003C/p>\n\u003Cp>Walk the whole JSON-LD graph, including nested nodes. Some shops wrap products in a \u003Ccode>ProductGroup\u003C/code> or an \u003Ccode>ItemList\u003C/code>, and a parser that only looks for a top-level \u003Ccode>Product\u003C/code> finds nothing. We made that mistake once on a big marketplace and nearly filed a working site as blocked.\u003C/p>\n\u003Ch2 id=\"step-4-probe-the-limits-before-you-build\">Step 4: Probe the Limits Before You Build\u003C/h2>\n\u003Cp>A working endpoint is the start. These tests decide how many requests a full catalogue will take.\u003C/p>\n\u003Cp>\u003Cstrong>Largest page size.\u003C/strong> Try 96, 250 and 500 items per page. One shop capped at 48. Another returned an error at 1,000 ids per call and worked at 500, which cut that site from 2,072 requests to 415 per market.\u003C/p>\n\u003Cp>\u003Cstrong>Past-the-end page.\u003C/strong> Request the page after the last one. If it is empty, paging is safe. If it returns products again, the site repeats pages, and you need a fixed list of page links instead of following "next".\u003C/p>\n\u003Cp>\u003Cstrong>Offset caps.\u003C/strong> Some search services refuse an offset beyond 10,000 results. Split the catalogue into bins, for example by category or id prefix, so each bin stays under the cap, and check that the bin totals add up to the site total.\u003C/p>\n\u003Cp>\u003Cstrong>Market and currency.\u003C/strong> Set the market with the URL, a header or a cookie that the endpoint accepts. We skipped one shop that set its market only through a form post that we could not replay.\u003C/p>\n\u003Cp>\u003Cstrong>Price check.\u003C/strong> Compare the API price with the shown page price on about 50 products per market. In one market the API price and the page price differed by a constant VAT factor, which one fixed column fixed. Some page-only discounts never reach any API, and we list those as a known limit.\u003C/p>\n\u003Ch2 id=\"step-5-read-the-json-carefully\">Step 5: Read the JSON Carefully\u003C/h2>\n\u003Cp>\u003Cstrong>One object per variant.\u003C/strong> If an endpoint gives you arrays such as \u003Ccode>sizes: [...]\u003C/code>, \u003Ccode>eans: [...]\u003C/code> and \u003Ccode>prices: [...]\u003C/code>, and one variant lacks an EAN, reading the arrays by position shifts every later EAN to the wrong size. Prefer an endpoint with one object per variant, where the size, the SKU and the EAN are fields of the same object. Test 5 rows against the page.\u003C/p>\n\u003Cp>\u003Cstrong>EANs as strings.\u003C/strong> A JSON number EAN can come out as \u003Ccode>4.006381333931E12\u003C/code>. Map it from a string field, then keep only digits with a cleaning rule.\u003C/p>\n\u003Cp>\u003Cstrong>Minor units.\u003C/strong> \u003Ccode>2495\u003C/code> with \u003Ccode>minor_unit: 2\u003C/code> is 24.95. Check the decimals field instead of assuming 2.\u003C/p>\n\u003Cp>\u003Cstrong>Codes hide in other fields.\u003C/strong> On one site the shop's SKU without its brand prefix was the maker's part number. It agreed with the EAN on 99.1% of rows where both existed. Test every code-like field, whatever it is called.\u003C/p>\n\u003Ch2 id=\"when-there-really-is-no-api\">When There Really Is No API\u003C/h2>\n\u003Cp>Some pages have no API, no JSON-LD and no microdata, or they load the price by a script that only runs in a browser. Then HTML selectors with a rendered page, or AI extraction, are the right tools. Just know what they cost: on our \u003Ca href=\"/pricing\">pricing\u003C/a> a plain page is €0.15 per 1,000, a browser page €0.75, and the hardest sites €3.75. Check that a plain fetch really fails before you pay for a browser. On one site a plain fetch gave exactly the same rows as a browser with residential proxies, at a 25th of the price.\u003C/p>\n\u003Cp>For the harder cases, see \u003Ca href=\"/blogs/how-to-scrape-javascript-ecommerce-websites-2026\">how to scrape JavaScript-heavy e-commerce sites\u003C/a>, and for why a 200 status is not proof you got the page, \u003Ca href=\"/blogs/http-200-not-success-eu-marketplaces-2026\">HTTP 200 is not success\u003C/a>.\u003C/p>\n\u003Cp>If you want an agent to do this hunting for you, the \u003Ca href=\"/blogs/build-web-scraper-with-claude-mcp-2026\">Claude and MCP walkthrough\u003C/a> shows the brief we give it: find the JSON source first, and use HTML only when there is none.\u003C/p>\n",{"title":11,"description":12,"badge":13,"benefits":14},"Frequently asked questions","Finding hidden shop APIs: questions answered","FAQ",[15,18,21,24,27],{"title":16,"description":17},"How do I find the API a shop page uses?","Open the page with the browser's Network panel, filter by Fetch/XHR, reload, and search the responses for a price you can see on the page. The response that contains it is usually the API.",{"title":19,"description":20},"Does Shopify have a public product JSON?","Yes. Shopify storefronts answer product and variant JSON, for example /products/handle.js, with price, before-price, weight, SKU, barcode and stock. Prices in the .js response are in cents.",{"title":22,"description":23},"Is JSON-LD good enough for price scraping?","Often, yes. It usually has name, price, currency and availability, and it rarely changes with the design. Walk the whole graph, because products can sit inside an ItemList or ProductGroup.",{"title":25,"description":26},"What should I test before trusting an API?","Test the largest page size, what happens past the last page, any offset cap, how the market is set, and whether the API price matches the page price on about 50 products.",{"title":28,"description":29},"Why is an API cheaper than scraping HTML?","One request often returns many products, and a plain fetch is enough. In our yarn project the JSON run cost about a ninth of the AI extraction run on the same links.","How to Find the Hidden JSON API Behind a Shop Page","Most shop pages load prices from JSON first. How we find those endpoints (Shopify, WooCommerce, Magento, JSON-LD, search APIs) and test them.","Raivo Kartau",null,"Scraping",6,[37,42,47],{"slug":38,"title":39,"image":40,"date":5,"category":34,"excerpt":41},"build-web-scraper-with-claude-mcp-2026","Build a Web Scraper with Claude and MCP: A Step-by-Step Walkthrough","/img/news/build-web-scraper-with-claude-mcp-2026.png","Connect Claude to Scrapewise through MCP and let it build, test and run price scrapers for you. The setup, the prompts and the guardrails we use.",{"slug":43,"title":44,"image":45,"date":5,"category":34,"excerpt":46},"llm-data-extraction-rules-do-the-maths-2026","LLM Data Extraction: Let the AI Read, Let Rules Do the Maths","/img/news/llm-data-extraction-rules-do-the-maths-2026.png","Our AI extraction got a price per 50 g right on 18 of 69 rows, and never by calculating. Moving the maths into rules fixed it on every row.",{"slug":48,"title":49,"image":50,"date":5,"category":34,"excerpt":51},"web-scraping-mistakes-price-scrapers-2026","14 Web Scraping Mistakes We Made Building Price Scrapers","/img/news/web-scraping-mistakes-price-scrapers-2026.png","Wrong sources, shifted EANs, Excel eating a brand name, AI doing sums. 14 mistakes from two real price monitoring builds, and the fix for each.",{"slug":53,"title":54},"http-200-not-success-eu-marketplaces-2026","HTTP 200 Doesn't Mean You Got the Page: 8 European Marketplaces Measured",{"slug":56,"title":57},"competitor-price-monitoring-case-study-2026","Competitor Price Monitoring Case Study: 100,000 SKUs, 88% Fewer Requests",[59,63,66,69,72,75,78],{"level":60,"text":61,"id":62},2,"Why Bother","why-bother",{"level":60,"text":64,"id":65},"Step 1: Watch What the Page Asks For","step-1-watch-what-the-page-asks-for",{"level":60,"text":67,"id":68},"Step 2: Try the Platform Endpoints","step-2-try-the-platform-endpoints",{"level":60,"text":70,"id":71},"Step 3: Don't Forget Structured Data in the HTML","step-3-don39t-forget-structured-data-in-the-html",{"level":60,"text":73,"id":74},"Step 4: Probe the Limits Before You Build","step-4-probe-the-limits-before-you-build",{"level":60,"text":76,"id":77},"Step 5: Read the JSON Carefully","step-5-read-the-json-carefully",{"level":60,"text":79,"id":80},"When There Really Is No API","when-there-really-is-no-api",[],1790769637105]