Analytics Regression Testing | Blue Frog Docs

Analytics Regression Testing

How to test analytics after deployments and prevent tracking regressions

Analytics Regression Testing

What This Means

Analytics regression testing is the practice of verifying that existing analytics tracking continues to work correctly after website changes, deployments, or updates. A regression occurs when previously working tracking breaks due to:

  • Site redesigns changing element selectors
  • CMS platform updates modifying page structure
  • Third-party script updates breaking compatibility
  • New features interfering with existing tags
  • Developer changes removing tracking code
  • Theme or template updates

Why it matters:

  • Tracking gaps lead to missing data and blind spots
  • Attribution models fail without complete data
  • Business decisions rely on accurate analytics
  • Regressions can go unnoticed for weeks or months
  • Historical comparisons become impossible

Common regression scenarios:

  • Button click tracking stops after CSS class changes
  • E-commerce tracking breaks after checkout redesign
  • Form submission events disappear after form builder update
  • Cross-domain tracking fails after subdomain launch
  • User ID tracking stops after authentication system change

How to Diagnose

Method 1: Before/After Comparison

When to use: Before and after major deployments.

Process:

  1. Pre-deployment baseline

    • Document all tracking currently working
    • Take screenshots of GTM Preview Mode showing tags firing
    • Export current GTM container as backup
    • Record GA4 events in DebugView

  2. Post-deployment validation

    • Run same test actions as baseline
    • Compare GTM Preview Mode results
    • Check for missing tags in "Tags Fired"
    • Verify same events appear in GA4 DebugView

  3. Compare data

    // Create checklist of critical tracking:
✓ Page views firing on all templates
✓ Add to cart button tracking
✓ Form submission events
✓ Purchase conversion tracking
✓ Video play events
✓ File download tracking
✓ Outbound link clicks
✓ Custom dimensions populated

Red flags:

  • Tags that previously fired now in "Not Fired"
  • Events missing from GA4 DebugView
  • Variables showing "undefined" that had values before
  • Console errors that weren't there before

Method 2: Automated Testing Frameworks

When to use: Regular deployments, continuous integration.

Tools:

  1. ObservePoint / Tag Inspector

    • Automated tag audits
    • Compare before/after scans
    • Alert on missing tags
    • Schedule regular checks

  2. Playwright / Puppeteer

    • Custom automated tests
    • Check for network requests
    • Verify data layer values

  3. Custom CI/CD Integration

    • Run tests before deployment
    • Fail build if tracking breaks

Example automated test:

// Playwright example test
const { test, expect } = require('@playwright/test');

test('GA4 page view fires on homepage', async ({ page }) => {
  // Set up request interception
  let ga4Fired = false;

  page.on('request', request => {
    const url = request.url();
    if (url.includes('google-analytics.com/g/collect')) {
      // Verify it's a page_view event
      if (url.includes('page_view')) {
        ga4Fired = true;
      }
    }
  });

  // Navigate to homepage
  await page.goto('https://www.yoursite.com');

  // Wait a bit for GA4 to fire
  await page.waitForTimeout(2000);

  // Assert GA4 fired
  expect(ga4Fired).toBeTruthy();
});

test('Add to cart tracking fires', async ({ page }) => {
  let addToCartFired = false;

  page.on('request', request => {
    const url = request.url();
    if (url.includes('google-analytics.com/g/collect')) {
      if (url.includes('add_to_cart')) {
        addToCartFired = true;
      }
    }
  });

  await page.goto('https://www.yoursite.com/products/test-product');
  await page.click('.add-to-cart-button');
  await page.waitForTimeout(2000);

  expect(addToCartFired).toBeTruthy();
});

test('Data layer contains transaction data', async ({ page }) => {
  await page.goto('https://www.yoursite.com/order-confirmation');

  // Check data layer
  const dataLayer = await page.evaluate(() => window.dataLayer);

  // Find purchase event
  const purchaseEvent = dataLayer.find(item =>
    item.event === 'purchase'
  );

  expect(purchaseEvent).toBeDefined();
  expect(purchaseEvent.transactionId).toBeDefined();
  expect(purchaseEvent.value).toBeGreaterThan(0);
});

Method 3: Production Monitoring

When to use: Ongoing monitoring for unexpected regressions.

Setup:

  1. GA4 Custom Alerts

    • Create alerts for critical metrics drops
    • Alert when event counts drop significantly
    • Monitor conversion rate changes

  2. Data anomaly detection

    • Set baseline for normal event volume
    • Alert when events drop below threshold
    • Track week-over-week consistency

  3. Manual spot checks

    • Weekly verification of critical tracking
    • Review Realtime report during business hours
    • Check for console errors in production

Example monitoring setup:

// Create GA4 custom insight for monitoring:
// Admin → Custom Insights (if available)
// Or use Google Sheets + GA4 API

// Example: Alert if add_to_cart events drop >20% day-over-day
// Set up in GA4 or external monitoring tool
// Metric: event_count (event_name = add_to_cart)
// Condition: Day-over-day change < -20%
// Alert: Send email to analytics team

Method 4: User Behavior Analysis

When to use: Catching subtle regressions that automated tests miss.

What to check:

  1. Session patterns

    • Are session durations normal?
    • Is bounce rate unchanged?
    • Are page depths similar?

  2. Conversion funnels

    • Are funnel drop-off rates consistent?
    • Did any funnel step break?
    • Are micro-conversions tracking?

  3. Event volume patterns

    • Compare event counts to previous weeks
    • Check for events that suddenly stopped
    • Look for unusual spikes (could indicate duplicates)

GA4 Analysis:

// Go to Explore → Funnel Exploration
// Compare last 7 days to previous 7 days
// Look for steps with dramatic changes

Step 1: View product (100%)
Step 2: Add to cart (40%) <- Was 45% last week
Step 3: Begin checkout (30%) <- Was 0% last week! (regression)
Step 4: Purchase (10%)

// Step 3 sudden drop to 0% indicates broken tracking

General Fixes

Fix 1: Automated Pre-Deployment Testing

Goal: Catch regressions before they reach production.

Implementation:

  1. Create test suite

    // tests/analytics-regression.spec.js
const criticalEvents = [
  { name: 'page_view', url: '/', trigger: 'load' },
  { name: 'add_to_cart', url: '/products/test', trigger: 'click', selector: '.add-to-cart' },
  { name: 'begin_checkout', url: '/cart', trigger: 'click', selector: '.checkout-button' },
  { name: 'purchase', url: '/test-purchase', trigger: 'load' }
];

criticalEvents.forEach(event => {
  test(`${event.name} fires correctly`, async ({ page }) => {
    // Test implementation
  });
});

  2. Add to CI/CD pipeline

    # .github/workflows/analytics-test.yml
name: Analytics Regression Tests

on:
  pull_request:
    branches: [ main ]

jobs:
  test-analytics:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install dependencies
        run: npm install
      - name: Run analytics tests
        run: npm run test:analytics
      - name: Fail if regression detected
        if: failure()
        run: exit 1

  3. Require passing tests before merge

    • Set up branch protection rules
    • Require analytics tests to pass
    • Block merges if tests fail

Fix 2: GTM Version Control and Rollback

Goal: Quickly rollback to working version if regression detected.

Process:

  1. Before deployment: Publish new GTM version

    // In GTM:
// 1. Submit workspace → Name it "Version 23 - Checkout redesign"
// 2. Publish to production
// 3. Note version number for rollback

  2. After deployment: Monitor for issues

    • Check GA4 Realtime report
    • Run manual test of critical paths
    • Watch for alerts

  3. If regression detected: Rollback immediately

    // In GTM:
// 1. Go to Admin → Versions
// 2. Find last known good version (Version 22)
// 3. Click Actions → Publish
// 4. Confirm publish to production
// Takes effect immediately (within minutes)

  4. Fix in workspace, then re-publish

    • Debug issue in GTM Preview Mode
    • Fix broken tags/triggers
    • Re-test
    • Publish new version

Best practice:

  • Name versions descriptively
  • Add version notes documenting changes
  • Keep track of what changed in each version
  • Test in staging environment first

Fix 3: Element Selector Stability

Goal: Make click tracking resistant to CSS class changes.

Problem:

// FRAGILE - Breaks if class name changes
Trigger Type: Click
Click Element: .blue-button-checkout-2024

// After redesign, class changes to:
// .checkout-cta-primary
// Tracking breaks!

Solution: Use stable selectors

// Method 1: Add data attributes for tracking
<button class="blue-button-checkout-2024" data-track="checkout-button">
  Checkout
</button>

// GTM Trigger:
// Click Element matches CSS selector: [data-track="checkout-button"]
// More stable - data-track won't change

// Method 2: Use multiple selector conditions
// GTM Trigger:
// Click Element contains text "Checkout" OR "Begin Checkout"
// AND Click Element is button

// Method 3: Use ID instead of class (if unique)
<button id="checkout-button" class="blue-button-checkout-2024">
// GTM Trigger: Click Element matches CSS selector #checkout-button

Implement across site:

<!-- Add tracking data attributes to all tracked elements -->
<button data-track="add-to-cart">Add to Cart</button>
<button data-track="begin-checkout">Proceed to Checkout</button>
<a href="/contact" data-track="contact-link">Contact Us</a>
<button data-track="newsletter-signup">Subscribe</button>

<!-- Document this pattern for developers -->
<!-- Include in style guide / component library -->

Fix 4: Data Layer Standardization

Goal: Prevent data layer regressions from code changes.

Problem:

// Developer changes data layer structure:
// Old format:
dataLayer.push({
  event: 'purchase',
  transactionId: '12345',
  transactionTotal: 99.99
});

// New format (after refactor):
dataLayer.push({
  event: 'purchase',
  transaction: {
    id: '12345',
    total: 99.99
  }
});

// GTM variables expecting transactionId now get undefined!

Solution: Create data layer specification

// 1. Document data layer structure:
// docs/data-layer-spec.md

/**
 * Purchase Event Data Layer
 * DO NOT CHANGE without updating GTM variables
 */
{
  event: 'purchase',          // Required, must be 'purchase'
  transactionId: string,      // Required, unique order ID
  transactionTotal: number,   // Required, total value
  items: [                    // Required, array of items
    {
      id: string,             // Required, product ID
      name: string,           // Required, product name
      price: number,          // Required, item price
      quantity: number        // Required, item quantity
    }
  ]
}

// 2. Add validation function:
function pushPurchase(data) {
  // Validate required fields
  if (!data.transactionId) {
    console.error('Missing transactionId in purchase event');
    return;
  }
  if (!data.transactionTotal) {
    console.error('Missing transactionTotal in purchase event');
    return;
  }

  // Push to data layer
  dataLayer.push({
    event: 'purchase',
    transactionId: data.transactionId,
    transactionTotal: data.transactionTotal,
    items: data.items || []
  });
}

// 3. Use validation function:
// Instead of direct dataLayer.push, use:
pushPurchase({
  transactionId: '12345',
  transactionTotal: 99.99,
  items: [...]
});

Fix 5: Monitoring Dashboard

Goal: Detect regressions quickly through automated monitoring.

Setup:

  1. Create monitoring dashboard

    // Google Sheets + GA4 API (example)
// Or use Looker Studio / Data Studio

// Track daily:
// - Total events by type
// - Conversion events count
// - Critical event counts

// Alert if:
// - Any event type drops >20% day-over-day
// - Zero events for critical event types
// - Unusual spikes (potential duplicates)

  2. Set up Slack/email alerts

    // Using Google Apps Script in Sheets:
function checkForRegressions() {
  var sheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName('Daily Events');

  // Get today's and yesterday's add_to_cart counts
  var today = sheet.getRange('B2').getValue();
  var yesterday = sheet.getRange('B3').getValue();

  var percentChange = ((today - yesterday) / yesterday) * 100;

  if (percentChange < -20) {
    // Send Slack alert
    var payload = {
      text: '⚠️ Analytics Regression Detected: add_to_cart events dropped ' +
            Math.abs(percentChange).toFixed(0) + '% today!'
    };

    UrlFetchApp.fetch('YOUR_SLACK_WEBHOOK_URL', {
      method: 'post',
      payload: JSON.stringify(payload)
    });
  }
}

// Run daily at 10am

  3. Daily manual check

    • Review dashboard every morning
    • Spot-check critical paths in production
    • Verify GTM Preview Mode weekly

Platform-Specific Guides

Platform Guide
Shopify Regression Testing on Shopify
WordPress Regression Testing on WordPress
Squarespace Regression Testing on Squarespace
Wix Regression Testing on Wix
Webflow Regression Testing on Webflow

Rollback Procedures

Emergency Rollback Checklist

When a critical regression is detected in production:

  1. Immediate actions (< 5 minutes)

    • Verify regression is real (not data delay)
    • Check if GTM or site deployment caused it
    • Alert stakeholders of tracking issue

  2. GTM rollback (5-10 minutes)

    • Log into Google Tag Manager
    • Go to Admin → Versions
    • Identify last known good version
    • Click Actions → Publish on that version
    • Verify tags firing in Preview Mode

  3. Site rollback (if GTM rollback doesn't fix)

    • Identify last deployment timestamp
    • Coordinate with dev team for rollback
    • Roll back to previous version
    • Verify tracking restored

  4. Post-incident (24 hours)

    • Document what broke and why
    • Identify root cause
    • Fix issue in staging environment
    • Add automated test to prevent recurrence
    • Re-deploy with fix

Further Reading

↑ Back to top
// SYS.FOOTER