[{"data":1,"prerenderedAt":127},["ShallowReactive",2],{"$fO3thLwswsjN_ggHiaMDfrfzvWDhkpF-Rk-B049fIhGY":3},{"title":4,"date":5,"dateModified":6,"datePublished":7,"dateModifiedISO":7,"image":8,"content":9,"faq":10,"metaTitle":11,"metaDescription":12,"author":13,"authorBio":10,"authorLinkedin":10,"authorTitle":10,"authorPhoto":14,"lastReviewed":10,"noindex":15,"researchBasis":10,"category":16,"readingTime":17,"related":18,"prev":36,"next":39,"toc":42,"takeaways":126},"Web Scraper Maintenance: Handling Site Changes & Technical Debt (2026)","02 Nov 2026","02 NOV 2026","2026-11-02","/img/news/web-scraper-maintenance-2026.svg","\u003Cp>A scraper is not a "set it and forget it" tool. It is a living system that requires feeding.\u003C/p>\n\u003Cp>Every website changes. HTML structure updates. JavaScript frameworks migrate. Anti-bot protections evolve. What works today breaks tomorrow.\u003C/p>\n\u003Cp>The difference between a scraper that runs reliably for years and one that breaks every month is \u003Cstrong>maintenance.\u003C/strong>\u003C/p>\n\u003Cp>This guide covers the operational side of web scraping: detecting breaks before they become crises, monitoring for site changes, and building a sustainable scraper operation.\u003C/p>\n\u003Ch2 id=\"why-scrapers-break-the-top-5-causes\">Why Scrapers Break: The Top 5 Causes\u003C/h2>\n\u003Ch3 id=\"1-site-redesigns-amp-dom-restructuring-30-of-breaks\">1. Site Redesigns & DOM Restructuring (30% of breaks)\u003C/h3>\n\u003Cp>The most common break. A website restructures their HTML:\u003C/p>\n\u003Cul>\n\u003Cli>Before: \u003Ccode><div class="product"><span class="price">$25</span></div>\u003C/code>\u003C/li>\n\u003Cli>After: \u003Ccode><div class="item"><span class="cost">$25</span></div>\u003C/code>\u003C/li>\n\u003C/ul>\n\u003Cp>Your selector \u003Ccode>.product .price\u003C/code> returns null. Data stops flowing.\u003C/p>\n\u003Cp>\u003Cstrong>Seasonal risk:\u003C/strong> Fashion, retail, and seasonal industries redesign quarterly (Spring/Summer, Fall/Winter). These breaks are predictable.\u003C/p>\n\u003Ch3 id=\"2-javascript-amp-framework-changes-25-of-breaks\">2. JavaScript & Framework Changes (25% of breaks)\u003C/h3>\n\u003Cp>A site migrates from jQuery to React. Or from React to Svelte. The page structure remains the same, but data loading mechanism changes:\u003C/p>\n\u003Cul>\n\u003Cli>Before: Data in HTML on page load\u003C/li>\n\u003Cli>After: Data loaded asynchronously, requires waiting\u003C/li>\n\u003C/ul>\n\u003Cp>Your curl-based scraper, which worked for 2 years, now gets empty pages.\u003C/p>\n\u003Ch3 id=\"3-anti-bot-protection-upgrades-20-of-breaks\">3. Anti-Bot Protection Upgrades (20% of breaks)\u003C/h3>\n\u003Cp>Websites upgrade anti-bot tools or detection:\u003C/p>\n\u003Cul>\n\u003Cli>Added Cloudflare: HTTP 403 Forbidden\u003C/li>\n\u003Cli>Added bot detection: Extra headers required\u003C/li>\n\u003Cli>Rate limits tightened: 429 errors start appearing\u003C/li>\n\u003Cli>IP blocking: Your datacenter IP is blocked\u003C/li>\n\u003C/ul>\n\u003Ch3 id=\"4-api-robotstxt-changes-15-of-breaks\">4. API / robots.txt Changes (15% of breaks)\u003C/h3>\n\u003Cp>The website updates robots.txt to disallow paths you depend on. Or they move an API endpoint. Or they add a Disallow rule specifically for your scraper patterns.\u003C/p>\n\u003Ch3 id=\"5-content-structure-changes-10-of-breaks\">5. Content Structure Changes (10% of breaks)\u003C/h3>\n\u003Cp>Product names moved from \u003Ccode><h1>\u003C/code> to \u003Ccode><h2>\u003C/code>. Prices moved from \u003Ccode>span\u003C/code> to \u003Ccode>data-price\u003C/code> attribute. Size options changed from dropdown to radio buttons.\u003C/p>\n\u003Chr>\n\u003Caside class=\"article__usecase-card\">\u003Cdiv class=\"article__usecase-label\">Related use case\u003C/div>\u003Ch3 class=\"article__usecase-title\">Any-site data scraper\u003C/h3>\u003Cp class=\"article__usecase-blurb\">No-code extraction from any website. Managed infrastructure, no anti-bot headaches.\u003C/p>\u003Ca class=\"article__usecase-link\" href=\"/use-cases/data-scraper\">See how it works →\u003C/a>\u003C/aside>\u003Ch2 id=\"strategy-1-automated-monitoring-detect-breaks-in-hours-not-d\">Strategy 1: Automated Monitoring (Detect Breaks in Hours, Not Days)\u003C/h2>\n\u003Cp>The goal is to know within 4 hours when a scraper breaks, not 3 days later.\u003C/p>\n\u003Ch3 id=\"pattern-1a-row-count-monitoring\">Pattern 1A: Row Count Monitoring\u003C/h3>\n\u003Cp>The simplest, most effective monitoring: \u003Cstrong>track the row count of every scraper run.\u003C/strong>\u003C/p>\n\u003Cdiv class=\"code-block\">\u003Cbutton type=\"button\" class=\"code-block__copy\" data-copy-code aria-label=\"Copy code\">Copy\u003C/button>\u003Cpre>\u003Ccode class=\"language-python\">def monitor_row_count(scraper_name, new_row_count, historical_average):\n  """Alert if row count deviates significantly"""\n\n  variance = (new_row_count - historical_average) / historical_average\n\n  if variance < -0.20:  # More than 20% drop\n    alert(severity="CRITICAL",\n          message=f"{scraper_name}: Row count dropped from {historical_average} to {new_row_count}")\n\n  elif variance < -0.10:  # 10-20% drop\n    alert(severity="WARNING",\n          message=f"{scraper_name}: Row count at {new_row_count} (usually {historical_average})")\n\n  # Track this run\n  log_row_count(scraper_name, new_row_count, datetime.now())\n\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>\u003Cstrong>Example:\u003C/strong>\u003C/p>\n\u003Cul>\n\u003Cli>Historical average: 1,000 products per run\u003C/li>\n\u003Cli>Tuesday run: 995 products (0.5% variance) → OK\u003C/li>\n\u003Cli>Wednesday run: 750 products (-25% variance) → ALERT\u003C/li>\n\u003Cli>Human inspection: Site redesigned. Selectors need updating.\u003C/li>\n\u003C/ul>\n\u003Ch3 id=\"pattern-1b-nullempty-field-monitoring\">Pattern 1B: Null/Empty Field Monitoring\u003C/h3>\n\u003Cp>Track which fields are returning null more often than usual.\u003C/p>\n\u003Cdiv class=\"code-block\">\u003Cbutton type=\"button\" class=\"code-block__copy\" data-copy-code aria-label=\"Copy code\">Copy\u003C/button>\u003Cpre>\u003Ccode class=\"language-python\">def monitor_field_quality(scraper_name, extracted_data):\n  """Alert if field null rate exceeds threshold"""\n\n  field_nulls = {\n    'price': sum(1 for item in extracted_data if not item.get('price')),\n    'title': sum(1 for item in extracted_data if not item.get('title')),\n    'inventory': sum(1 for item in extracted_data if not item.get('inventory')),\n  }\n\n  for field, null_count in field_nulls.items():\n    null_rate = null_count / len(extracted_data)\n\n    if null_rate > 0.10:  # More than 10% nulls\n      alert(severity="CRITICAL",\n            message=f"{scraper_name}: {field} is null in {null_rate * 100}% of rows")\n\u003C/code>\u003C/pre>\u003C/div>\n\u003Ch3 id=\"pattern-1c-http-status-code-monitoring\">Pattern 1C: HTTP Status Code Monitoring\u003C/h3>\n\u003Cp>Track HTTP response codes across all scraper runs.\u003C/p>\n\u003Cdiv class=\"code-block\">\u003Cbutton type=\"button\" class=\"code-block__copy\" data-copy-code aria-label=\"Copy code\">Copy\u003C/button>\u003Cpre>\u003Ccode class=\"language-python\">def monitor_http_health(scraper_name, http_codes):\n  """Alert on unexpected status codes"""\n\n  expected_codes = {200}  # Usually just 200\n  unexpected = [code for code in http_codes if code not in expected_codes]\n\n  if 403 in http_codes:\n    alert(severity="CRITICAL", message="403 Forbidden: IP may be blocked")\n\n  if 429 in http_codes:\n    alert(severity="CRITICAL", message="429 Rate Limited: Reduce request rate")\n\n  if sum(1 for c in http_codes if c >= 500) > 5:\n    alert(severity="WARNING", message="Multiple 500 errors: Site may be down")\n\u003C/code>\u003C/pre>\u003C/div>\n\u003Ch3 id=\"building-a-monitoring-dashboard\">Building a Monitoring Dashboard\u003C/h3>\n\u003Cp>Track all scrapers in one place:\u003C/p>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Scraper\u003C/th>\n\u003Cth>Last Run\u003C/th>\n\u003Cth>Row Count\u003C/th>\n\u003Cth>Status\u003C/th>\n\u003Cth>Hours Since Update\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>H&M Prices\u003C/td>\n\u003Ctd>2h ago\u003C/td>\n\u003Ctd>2,451\u003C/td>\n\u003Ctd>✓ OK\u003C/td>\n\u003Ctd>2\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>ASOS Inventory\u003C/td>\n\u003Ctd>4h ago\u003C/td>\n\u003Ctd>1,245\u003C/td>\n\u003Ctd>⚠️ LOW\u003C/td>\n\u003Ctd>4\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Zara Trends\u003C/td>\n\u003Ctd>Failed\u003C/td>\n\u003Ctd>-\u003C/td>\n\u003Ctd>❌ DOWN\u003C/td>\n\u003Ctd>24\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>Amazon Sellers\u003C/td>\n\u003Ctd>1h ago\u003C/td>\n\u003Ctd>5,623\u003C/td>\n\u003Ctd>✓ OK\u003C/td>\n\u003Ctd>1\u003C/td>\n\u003C/tr>\n\u003C/tbody>\u003C/table>\n\u003Cp>\u003Cstrong>Alert thresholds:\u003C/strong>\u003C/p>\n\u003Cul>\n\u003Cli>🔴 Critical: No data (> 24h since last run) or 0 rows returned\u003C/li>\n\u003Cli>🟠 Warning: 10–20% drop in row count, runs older than 12 hours\u003C/li>\n\u003Cli>🟢 OK: Expected row count range, runs within last 6 hours\u003C/li>\n\u003C/ul>\n\u003Chr>\n\u003Ch2 id=\"strategy-2-automated-testing-catch-breaks-before-production\">Strategy 2: Automated Testing (Catch Breaks Before Production)\u003C/h2>\n\u003Cp>Before deploying a scraper, verify it works. Test again regularly.\u003C/p>\n\u003Ch3 id=\"test-pattern-1-sample-data-validation\">Test Pattern 1: Sample Data Validation\u003C/h3>\n\u003Cp>Scrape a known set of URLs. Verify the extracted data is correct.\u003C/p>\n\u003Cdiv class=\"code-block\">\u003Cbutton type=\"button\" class=\"code-block__copy\" data-copy-code aria-label=\"Copy code\">Copy\u003C/button>\u003Cpre>\u003Ccode class=\"language-python\">def test_scraper_accuracy():\n  """Test against known products with known data"""\n\n  test_cases = [\n    {\n      'url': 'https://example.com/product/123',\n      'expected_title': 'T-Shirt Blue',\n      'expected_price': 25.00,\n    },\n    {\n      'url': 'https://example.com/product/456',\n      'expected_title': 'Jeans Black',\n      'expected_price': 75.00,\n    },\n  ]\n\n  for test_case in test_cases:\n    result = scrape_product(test_case['url'])\n\n    assert result['title'] == test_case['expected_title'], \\\n      f"Title mismatch. Got {result['title']}"\n\n    assert result['price'] == test_case['expected_price'], \\\n      f"Price mismatch. Got {result['price']}"\n\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>Run this test:\u003C/p>\n\u003Cul>\n\u003Cli>After every code change\u003C/li>\n\u003Cli>Weekly on production data (automated)\u003C/li>\n\u003Cli>After you detect a site redesign\u003C/li>\n\u003C/ul>\n\u003Ch3 id=\"test-pattern-2-selector-stability-test\">Test Pattern 2: Selector Stability Test\u003C/h3>\n\u003Cp>Verify selectors still find elements.\u003C/p>\n\u003Cdiv class=\"code-block\">\u003Cbutton type=\"button\" class=\"code-block__copy\" data-copy-code aria-label=\"Copy code\">Copy\u003C/button>\u003Cpre>\u003Ccode class=\"language-python\">def test_selectors(page_html, selectors):\n  """Verify selectors find expected elements"""\n\n  from bs4 import BeautifulSoup\n  soup = BeautifulSoup(page_html, 'html.parser')\n\n  for field, selector in selectors.items():\n    elements = soup.select(selector)\n    assert len(elements) > 0, f"Selector {field} found no elements"\n\u003C/code>\u003C/pre>\u003C/div>\n\u003Ch3 id=\"test-pattern-3-regression-testing\">Test Pattern 3: Regression Testing\u003C/h3>\n\u003Cp>After a scraper break is fixed, test that the fix works and does not break other fields.\u003C/p>\n\u003Cdiv class=\"code-block\">\u003Cbutton type=\"button\" class=\"code-block__copy\" data-copy-code aria-label=\"Copy code\">Copy\u003C/button>\u003Cpre>\u003Ccode class=\"language-python\">def test_regression():\n  """Before: price scraping broken. After: price scraping fixed."""\n\n  # Setup: Get a product page that caused the original failure\n  url = 'https://example.com/product-that-broke-scraper'\n  result = scrape_product(url)\n\n  # Test 1: Price is now extracted (the bug that was fixed)\n  assert result['price'] is not None, "Price is still null"\n  assert result['price'] > 0, "Price is 0 or negative"\n\n  # Test 2: Other fields did not break (regression)\n  assert result['title'] is not None, "Title broke when fixing price"\n  assert result['inventory'] is not None, "Inventory broke when fixing price"\n\u003C/code>\u003C/pre>\u003C/div>\n\u003Chr>\n\u003Caside class=\"article__inline-cta\">\u003Cp class=\"article__inline-cta-text\">Try ScrapeWise on your own URL. \u003Cstrong>Your first 5 requests are free.\u003C/strong>\u003C/p>\u003Ca class=\"article__inline-cta-btn\" href=\"https://portal.scrapewise.ai/register\" target=\"_blank\" rel=\"noopener\">Start Free →\u003C/a>\u003C/aside>\u003Ch2 id=\"strategy-3-version-control-for-scraper-configs\">Strategy 3: Version Control for Scraper Configs\u003C/h2>\n\u003Cp>Treat your scraper configurations like code.\u003C/p>\n\u003Cdiv class=\"code-block\">\u003Cbutton type=\"button\" class=\"code-block__copy\" data-copy-code aria-label=\"Copy code\">Copy\u003C/button>\u003Cpre>\u003Ccode class=\"language-yaml\"># scraper_config_v3.2.yaml\nname: "H&M Product Prices"\nversion: "3.2"\nupdated: "2026-11-01"\n\nselectors:\n  title: "[data-product-title]"  # Changed from .product-name in v3.1\n  price: "[data-price]"\n  availability: "[data-in-stock]"\n\n# Fallback selectors if primary fails\nfallback_selectors:\n  title: "h1"\n  price: ".cost"\n\nchange_log:\n  v3.2:\n    date: "2026-11-01"\n    reason: "Site redesign 2026-11-01"\n    changed: "title selector (.product-name → [data-product-title])"\n\n  v3.1:\n    date: "2026-09-15"\n    reason: "Anti-bot protection upgrade"\n    changed: "Added User-Agent rotation, retry logic"\n\n  v3.0:\n    date: "2026-08-01"\n    reason: "Initial release"\n\u003C/code>\u003C/pre>\u003C/div>\n\u003Cp>\u003Cstrong>Benefits:\u003C/strong>\u003C/p>\n\u003Cul>\n\u003Cli>Rollback to previous config if a change breaks things\u003C/li>\n\u003Cli>See what changed and when\u003C/li>\n\u003Cli>Explain why a change was needed\u003C/li>\n\u003C/ul>\n\u003Chr>\n\u003Ch2 id=\"strategy-4-change-detection-know-when-a-site-changed\">Strategy 4: Change Detection (Know When a Site Changed)\u003C/h2>\n\u003Cp>Automatically detect when a website has changed so you can act proactively.\u003C/p>\n\u003Ch3 id=\"detection-pattern-1-html-hash-monitoring\">Detection Pattern 1: HTML Hash Monitoring\u003C/h3>\n\u003Cdiv class=\"code-block\">\u003Cbutton type=\"button\" class=\"code-block__copy\" data-copy-code aria-label=\"Copy code\">Copy\u003C/button>\u003Cpre>\u003Ccode class=\"language-python\">import hashlib\nfrom datetime import datetime\n\ndef detect_site_changes():\n  """Detect if site structure changed by comparing HTML hash"""\n\n  url = 'https://example.com/product/123'\n  response = requests.get(url)\n  current_hash = hashlib.md5(response.text.encode()).hexdigest()\n\n  # Get previous hash\n  previous_hash = get_stored_hash(url)\n\n  if current_hash != previous_hash:\n    alert(severity="WARNING",\n          message=f"Site structure changed for {url}")\n\n    # Store new hash for next comparison\n    store_hash(url, current_hash, datetime.now())\n\u003C/code>\u003C/pre>\u003C/div>\n\u003Ch3 id=\"detection-pattern-2-selector-validation-on-each-run\">Detection Pattern 2: Selector Validation on Each Run\u003C/h3>\n\u003Cdiv class=\"code-block\">\u003Cbutton type=\"button\" class=\"code-block__copy\" data-copy-code aria-label=\"Copy code\">Copy\u003C/button>\u003Cpre>\u003Ccode class=\"language-python\">def validate_selectors_on_run(page_html, selectors):\n  """Before scraping, verify selectors still work"""\n\n  from bs4 import BeautifulSoup\n  soup = BeautifulSoup(page_html, 'html.parser')\n\n  broken_selectors = []\n\n  for field, selector in selectors.items():\n    if not soup.select(selector):\n      broken_selectors.append(field)\n\n  if broken_selectors:\n    alert(severity="CRITICAL",\n          message=f"Selectors broken: {broken_selectors}. Site may have redesigned.")\n    # Stop scraping and wait for manual fix\n    return False\n\n  return True  # Proceed with scraping\n\u003C/code>\u003C/pre>\u003C/div>\n\u003Chr>\n\u003Ch2 id=\"the-4-step-repair-process\">The 4-Step Repair Process\u003C/h2>\n\u003Cp>When a scraper breaks, follow this process:\u003C/p>\n\u003Ch3 id=\"step-1-identify-the-break-30-minutes\">Step 1: Identify the Break (30 minutes)\u003C/h3>\n\u003Col>\n\u003Cli>Review alert: What broke? (row count, specific field, HTTP error?)\u003C/li>\n\u003Cli>Manual inspection: Visit the site in a browser. Did the layout change?\u003C/li>\n\u003Cli>Check logs: What error is the scraper reporting?\u003C/li>\n\u003C/ol>\n\u003Ch3 id=\"step-2-assess-impact-15-minutes\">Step 2: Assess Impact (15 minutes)\u003C/h3>\n\u003Col>\n\u003Cli>How many scraper runs are affected? (just today, or 3 days?)\u003C/li>\n\u003Cli>How critical is the data? (pricing: critical. reviews: less critical)\u003C/li>\n\u003Cli>Is there a workaround? (API, different URL, different field?)\u003C/li>\n\u003C/ol>\n\u003Cp>\u003Cstrong>Decision tree:\u003C/strong>\u003C/p>\n\u003Cul>\n\u003Cli>Critical impact + no workaround → Escalate to engineer (now)\u003C/li>\n\u003Cli>Critical impact + workaround exists → Use workaround temporarily\u003C/li>\n\u003Cli>Non-critical impact → Schedule fix for next sprint\u003C/li>\n\u003C/ul>\n\u003Ch3 id=\"step-3-root-cause-14-hours\">Step 3: Root Cause (1–4 hours)\u003C/h3>\n\u003Cp>Use a debugger to understand what broke:\u003C/p>\n\u003Cdiv class=\"code-block\">\u003Cbutton type=\"button\" class=\"code-block__copy\" data-copy-code aria-label=\"Copy code\">Copy\u003C/button>\u003Cpre>\u003Ccode class=\"language-python\">from selenium import webdriver\n\ndriver = webdriver.Chrome()\ndriver.get('https://example.com/product/123')\n\n# Take screenshot of current page\ndriver.save_screenshot('current_page.png')\n\n# Test each selector individually\nelements = driver.find_elements('css selector', '.product-price')\nprint(f"Found {len(elements)} price elements")\n\n# If 0, the selector is wrong. Find the correct selector.\n# Right-click in browser > Inspect > find the price element > copy selector\n\u003C/code>\u003C/pre>\u003C/div>\n\u003Ch3 id=\"step-4-deploy-fix-12-hours\">Step 4: Deploy Fix (1–2 hours)\u003C/h3>\n\u003Col>\n\u003Cli>\u003Cstrong>Update selector\u003C/strong> to the new location\u003C/li>\n\u003Cli>\u003Cstrong>Add fallback selectors\u003C/strong> (2–3 alternatives)\u003C/li>\n\u003Cli>\u003Cstrong>Test\u003C/strong> against 5–10 products (manual spot-check)\u003C/li>\n\u003Cli>\u003Cstrong>Deploy\u003C/strong> and monitor for 24h\u003C/li>\n\u003C/ol>\n\u003Chr>\n\u003Ch2 id=\"real-example-hampm-redesign-incident\">Real Example: H&M Redesign Incident\u003C/h2>\n\u003Cp>\u003Cstrong>Timeline:\u003C/strong>\u003C/p>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Time\u003C/th>\n\u003Cth>Event\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>2026-11-01 08:00\u003C/td>\n\u003Ctd>H&M launches new website design\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2026-11-01 08:15\u003C/td>\n\u003Ctd>Scraper runs, returns 1,200 products (normal)\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2026-11-01 10:00\u003C/td>\n\u003Ctd>New page layout fully rolled out\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2026-11-01 12:00\u003C/td>\n\u003Ctd>Scraper runs again, returns 0 products\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2026-11-01 12:05\u003C/td>\n\u003Ctd>Monitoring system alerts: "Row count: 0"\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2026-11-01 12:30\u003C/td>\n\u003Ctd>Engineer reviews logs, finds selector error\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2026-11-01 12:45\u003C/td>\n\u003Ctd>Engineer visits site, inspects new HTML structure\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2026-11-01 13:15\u003C/td>\n\u003Ctd>New selectors updated in config\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2026-11-01 13:30\u003C/td>\n\u003Ctd>Scraper tested on 10 products, verified correct\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2026-11-01 14:00\u003C/td>\n\u003Ctd>Fix deployed, scraper runs, returns 1,200 products\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>2026-11-01 14:05\u003C/td>\n\u003Ctd>Pricing team confirms data is flowing again\u003C/td>\n\u003C/tr>\n\u003C/tbody>\u003C/table>\n\u003Cp>\u003Cstrong>Total downtime:\u003C/strong> 2 hours\n\u003Cstrong>Cost:\u003C/strong> 1 hour of engineer time\n\u003Cstrong>If monitoring was not in place:\u003C/strong> Would have discovered 24h later (missing an entire day of pricing data)\u003C/p>\n\u003Chr>\n\u003Ch2 id=\"building-a-sustainable-scraping-operation\">Building a Sustainable Scraping Operation\u003C/h2>\n\u003Cp>To run scrapers reliably at scale:\u003C/p>\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Component\u003C/th>\n\u003Cth>Purpose\u003C/th>\n\u003Cth>Effort\u003C/th>\n\u003C/tr>\n\u003C/thead>\n\u003Ctbody>\u003Ctr>\n\u003Ctd>\u003Cstrong>Monitoring Dashboard\u003C/strong>\u003C/td>\n\u003Ctd>Know when scrapers break\u003C/td>\n\u003Ctd>16 hours\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>\u003Cstrong>Automated Tests\u003C/strong>\u003C/td>\n\u003Ctd>Catch breaks before production\u003C/td>\n\u003Ctd>12 hours\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>\u003Cstrong>Configuration Management\u003C/strong>\u003C/td>\n\u003Ctd>Track changes, enable rollbacks\u003C/td>\n\u003Ctd>8 hours\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>\u003Cstrong>Change Detection\u003C/strong>\u003C/td>\n\u003Ctd>Proactive alerts about site changes\u003C/td>\n\u003Ctd>12 hours\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>\u003Cstrong>Runbook\u003C/strong>\u003C/td>\n\u003Ctd>Documented repair process\u003C/td>\n\u003Ctd>4 hours\u003C/td>\n\u003C/tr>\n\u003Ctr>\n\u003Ctd>\u003Cstrong>Alerting\u003C/strong>\u003C/td>\n\u003Ctd>Slack/Email notifications\u003C/td>\n\u003Ctd>4 hours\u003C/td>\n\u003C/tr>\n\u003C/tbody>\u003C/table>\n\u003Cp>\u003Cstrong>Total setup investment:\u003C/strong> ~56 hours\n\u003Cstrong>Time saved per scraper break:\u003C/strong> 8–16 hours (debugging + fix + testing)\n\u003Cstrong>Break frequency:\u003C/strong> 1–2 per scraper per year (depends on site volatility)\n\u003Cstrong>ROI:\u003C/strong> Breaks will happen. Detection + monitoring reduce cost 5–10×\u003C/p>\n\u003Chr>\n\u003Ch2 id=\"conclusion-maintenance-is-the-unglamorous-secret-to-reliable\">Conclusion: Maintenance Is the Unglamorous Secret to Reliable Scraping\u003C/h2>\n\u003Cp>The companies with the most reliable scrapers are not using fancy AI or novel techniques. They are using monitoring, testing, and a documented repair process.\u003C/p>\n\u003Cp>Invest in operations. Your future self will thank you.\u003C/p>\n\u003Cp>\u003Cstrong>Ready to set up maintenance systems?\u003C/strong> Try \u003Ca href=\"/demo\">ScrapeWise's built-in monitoring and alerting\u003C/a> — automatic break detection and repair suggestions.\u003C/p>\n",null,"Scraper Maintenance & Monitoring: Detect Breaks Before They Happen","Web scraper maintenance guide. How to detect when scrapers break, monitor for site changes, repair strategy, and build sustainable scraper operations.","Siim Brazier","/img/team/siim.jpg",false,"Insights",8,[19,25,31],{"slug":20,"title":21,"image":22,"date":23,"category":16,"excerpt":24},"http-200-not-success-eu-marketplaces-2026","HTTP 200 Doesn't Mean You Got the Page: 8 European Marketplaces Measured","/img/news/http-200-not-success-eu-marketplaces-2026.png","30 September 2026","We fetched 8 European marketplaces in September 2026. Four answered HTTP 200 and only two returned the page. What the other bodies actually contained.",{"slug":26,"title":27,"image":28,"date":29,"category":16,"excerpt":30},"export-google-maps-to-excel-csv-2026","Export Google Maps Results to Excel or CSV: A List of Businesses by City","/img/news/export-google-maps-to-excel-csv-2026.png","15 Sep 2026","Google Maps has no export button. See 3 ways to get a list of businesses by city into Excel or CSV, what each exports, and where Google's terms stop you.",{"slug":32,"title":33,"image":34,"date":29,"category":16,"excerpt":35},"youtube-scraper-api-python-2026","YouTube Scraper API with Python: Official API vs Scraper, With Working Code","/img/news/youtube-scraper-api-python-2026.png","Collect YouTube channel data in Python two ways: the official API (1 unit per lookup, 10,000 free daily) or a scraper API. Working code and EU rules.",{"slug":37,"title":38},"web-scraping-roi-calculator-business-value-2026","Web Scraping ROI Calculator: Measure the Business Value of Data Extraction (2026)",{"slug":40,"title":41},"scraping-fashion-ecommerce-inventory-tracking-2026","Scraping Fashion E-commerce Sites: Inventory Tracking & Competitive Analysis (2026)",[43,47,51,54,57,60,63,66,69,72,75,78,81,84,87,90,93,96,99,102,105,108,111,114,117,120,123],{"level":44,"text":45,"id":46},2,"Why Scrapers Break: The Top 5 Causes","why-scrapers-break-the-top-5-causes",{"level":48,"text":49,"id":50},3,"1. Site Redesigns & DOM Restructuring (30% of breaks)","1-site-redesigns-amp-dom-restructuring-30-of-breaks",{"level":48,"text":52,"id":53},"2. JavaScript & Framework Changes (25% of breaks)","2-javascript-amp-framework-changes-25-of-breaks",{"level":48,"text":55,"id":56},"3. Anti-Bot Protection Upgrades (20% of breaks)","3-anti-bot-protection-upgrades-20-of-breaks",{"level":48,"text":58,"id":59},"4. API / robots.txt Changes (15% of breaks)","4-api-robotstxt-changes-15-of-breaks",{"level":48,"text":61,"id":62},"5. Content Structure Changes (10% of breaks)","5-content-structure-changes-10-of-breaks",{"level":44,"text":64,"id":65},"Strategy 1: Automated Monitoring (Detect Breaks in Hours, Not Days)","strategy-1-automated-monitoring-detect-breaks-in-hours-not-d",{"level":48,"text":67,"id":68},"Pattern 1A: Row Count Monitoring","pattern-1a-row-count-monitoring",{"level":48,"text":70,"id":71},"Pattern 1B: Null/Empty Field Monitoring","pattern-1b-nullempty-field-monitoring",{"level":48,"text":73,"id":74},"Pattern 1C: HTTP Status Code Monitoring","pattern-1c-http-status-code-monitoring",{"level":48,"text":76,"id":77},"Building a Monitoring Dashboard","building-a-monitoring-dashboard",{"level":44,"text":79,"id":80},"Strategy 2: Automated Testing (Catch Breaks Before Production)","strategy-2-automated-testing-catch-breaks-before-production",{"level":48,"text":82,"id":83},"Test Pattern 1: Sample Data Validation","test-pattern-1-sample-data-validation",{"level":48,"text":85,"id":86},"Test Pattern 2: Selector Stability Test","test-pattern-2-selector-stability-test",{"level":48,"text":88,"id":89},"Test Pattern 3: Regression Testing","test-pattern-3-regression-testing",{"level":44,"text":91,"id":92},"Strategy 3: Version Control for Scraper Configs","strategy-3-version-control-for-scraper-configs",{"level":44,"text":94,"id":95},"Strategy 4: Change Detection (Know When a Site Changed)","strategy-4-change-detection-know-when-a-site-changed",{"level":48,"text":97,"id":98},"Detection Pattern 1: HTML Hash Monitoring","detection-pattern-1-html-hash-monitoring",{"level":48,"text":100,"id":101},"Detection Pattern 2: Selector Validation on Each Run","detection-pattern-2-selector-validation-on-each-run",{"level":44,"text":103,"id":104},"The 4-Step Repair Process","the-4-step-repair-process",{"level":48,"text":106,"id":107},"Step 1: Identify the Break (30 minutes)","step-1-identify-the-break-30-minutes",{"level":48,"text":109,"id":110},"Step 2: Assess Impact (15 minutes)","step-2-assess-impact-15-minutes",{"level":48,"text":112,"id":113},"Step 3: Root Cause (1–4 hours)","step-3-root-cause-14-hours",{"level":48,"text":115,"id":116},"Step 4: Deploy Fix (1–2 hours)","step-4-deploy-fix-12-hours",{"level":44,"text":118,"id":119},"Real Example: H&M Redesign Incident","real-example-hampm-redesign-incident",{"level":44,"text":121,"id":122},"Building a Sustainable Scraping Operation","building-a-sustainable-scraping-operation",{"level":44,"text":124,"id":125},"Conclusion: Maintenance Is the Unglamorous Secret to Reliable Scraping","conclusion-maintenance-is-the-unglamorous-secret-to-reliable",[],1793646897881]