Webhook Delivery Failures

What This Means

Webhook delivery failures occur when external services cannot successfully send event notifications to your application's webhook endpoints. These failures prevent your application from receiving real-time updates about important events, breaking automation, causing data gaps, and degrading user experience.

Impact on Your Business

Missed Events:

• Payment confirmations not received
• Order updates delayed
• User actions not tracked
• Inventory changes missed
• Critical notifications lost

Data Inconsistency:

• Out-of-sync records
• Stale information displayed
• Duplicate data processing
• Missing transaction records
• Incomplete audit trails

Broken Automation:

• Workflows don't trigger
• Email notifications not sent
• Integrations fail silently
• Business processes stall
• Customer experience degraded

Common Causes

Endpoint Issues:

• URL not publicly accessible
• SSL certificate invalid or expired
• Server not responding (timeouts)
• Wrong HTTP response codes
• Endpoint moved or deleted

Configuration Errors:

• Webhook URL misconfigured
• Wrong HTTP method (GET vs POST)
• Missing or invalid signature verification
• Payload format mismatch
• Content-Type header issues

Network/Infrastructure:

• Firewall blocking requests
• Load balancer misconfiguration
• Rate limiting on your server
• DDoS protection blocking legitimate webhooks
• DNS resolution failures

How to Diagnose

Method 1: Check Webhook Dashboard

Most services provide webhook monitoring dashboards:

1. Login to service provider (Stripe, Shopify, etc.)
2. Navigate to Webhooks or Developers section
3. Check webhook delivery status
4. Review failed deliveries
5. Check error messages and response codes

What to Look For:

Status: Failed
Response Code: 502 Bad Gateway
Error: Connection timeout after 10s
Attempts: 3 of 3 (retries exhausted)
Last Attempt: 2024-01-15 14:32:01 UTC

Method 2: Test Webhook Endpoint Manually

Test your endpoint using curl or Postman:

# Test basic connectivity
curl -X POST https://your-domain.com/webhooks/stripe \
  -H "Content-Type: application/json" \
  -d '{"test": "data"}'

# Test with signature (example for Stripe)
curl -X POST https://your-domain.com/webhooks/stripe \
  -H "Content-Type: application/json" \
  -H "Stripe-Signature: t=timestamp,v1=signature" \
  -d '{"id": "evt_test", "type": "charge.succeeded"}'

# Check response
# Should return 200 OK
# Response body should acknowledge receipt

What to Look For:

• HTTP 200 OK response (success)
• HTTP 401/403 (authentication issues)
• HTTP 404 (endpoint not found)
• HTTP 500 (server error)
• Timeout (no response)
• SSL/TLS errors

Method 3: Review Server Logs

Check your application and web server logs:

# Application logs
tail -f /var/log/app/webhooks.log

# Nginx logs
tail -f /var/log/nginx/access.log
tail -f /var/log/nginx/error.log

# Apache logs
tail -f /var/log/apache2/access.log
tail -f /var/log/apache2/error.log

Common log patterns:

[ERROR] Webhook signature verification failed
[WARN] Webhook received but processing timed out
[ERROR] Database connection failed during webhook processing
[INFO] Webhook received: order.created (id: 12345)
[ERROR] Invalid JSON payload in webhook

Method 4: Use Webhook Testing Tools

Test webhooks using dedicated tools:

Tools:

• webhook.site - Inspect webhook payloads
• RequestBin - Capture and inspect requests
• ngrok - Test localhost endpoints publicly
• Service-specific testing (Stripe CLI, Shopify CLI)

Process:

1. Create test endpoint URL
2. Configure webhook to point to test URL
3. Trigger test event
4. Inspect received payload
5. Verify headers, body, signature

Method 5: Check SSL Certificate

Verify your SSL certificate is valid:

# Check SSL certificate
openssl s_client -connect your-domain.com:443 -servername your-domain.com

# Check certificate expiration
echo | openssl s_client -servername your-domain.com \
  -connect your-domain.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# Verify certificate chain
curl -vI https://your-domain.com/webhooks/endpoint

What to Look For:

• Certificate expired
• Self-signed certificate
• Certificate mismatch (wrong domain)
• Incomplete certificate chain
• Weak cipher suites

General Fixes

Fix 1: Configure Webhook Endpoint Correctly

Ensure your webhook endpoint is properly set up:

Node.js/Express example:

const express = require('express');
const app = express();

// Raw body parser for signature verification
app.use('/webhooks', express.raw({ type: 'application/json' }));

// Webhook endpoint
app.post('/webhooks/stripe', async (req, res) => {
  // Get raw body for signature verification
  const payload = req.body;
  const signature = req.headers['stripe-signature'];

  try {
    // Verify webhook signature
    const event = stripe.webhooks.constructEvent(
      payload,
      signature,
      process.env.STRIPE_WEBHOOK_SECRET
    );

    // Process event
    await processWebhookEvent(event);

    // Respond quickly (within 10 seconds)
    res.status(200).json({ received: true });

    // Process event asynchronously if needed
    // Don't make webhook wait for slow operations

  } catch (err) {
    console.error('Webhook signature verification failed:', err.message);
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }
});

async function processWebhookEvent(event) {
  switch (event.type) {
    case 'payment_intent.succeeded':
      await handlePaymentSuccess(event.data.object);
      break;
    case 'customer.subscription.deleted':
      await handleSubscriptionCanceled(event.data.object);
      break;
    default:
      console.log(`Unhandled event type: ${event.type}`);
  }
}

app.listen(3000);

Important considerations:

• Return 200 OK quickly (< 10 seconds)
• Process heavy work asynchronously (queues)
• Use raw body parser for signature verification
• Log all webhook receipts
• Handle errors gracefully

Fix 2: Implement Webhook Signature Verification

Verify webhooks are from legitimate sources:

Stripe example:

const stripe = require('stripe')(process.env.STRIPE_SECRET_KEY);

function verifyStripeWebhook(payload, signature, secret) {
  try {
    const event = stripe.webhooks.constructEvent(
      payload,
      signature,
      secret
    );
    return event;
  } catch (err) {
    console.error('Signature verification failed:', err.message);
    throw new Error('Invalid signature');
  }
}

// In webhook handler
app.post('/webhooks/stripe', (req, res) => {
  const signature = req.headers['stripe-signature'];

  try {
    const event = verifyStripeWebhook(
      req.body,
      signature,
      process.env.STRIPE_WEBHOOK_SECRET
    );

    // Signature valid, process event
    processEvent(event);
    res.status(200).json({ received: true });

  } catch (err) {
    res.status(400).send(`Webhook Error: ${err.message}`);
  }
});

Shopify example:

const crypto = require('crypto');

function verifyShopifyWebhook(payload, hmacHeader, secret) {
  const hash = crypto
    .createHmac('sha256', secret)
    .update(payload)
    .digest('base64');

  if (hash !== hmacHeader) {
    throw new Error('Invalid webhook signature');
  }
}

app.post('/webhooks/shopify', (req, res) => {
  const hmac = req.headers['x-shopify-hmac-sha256'];

  try {
    verifyShopifyWebhook(
      req.body,
      hmac,
      process.env.SHOPIFY_WEBHOOK_SECRET
    );

    // Process webhook
    res.status(200).send('OK');

  } catch (err) {
    res.status(401).send('Unauthorized');
  }
});

Fix 3: Implement Retry Logic and Idempotency

Handle duplicate deliveries and implement retries:

// Track processed webhooks to handle duplicates
const processedWebhooks = new Set();

app.post('/webhooks/endpoint', async (req, res) => {
  const webhookId = req.headers['x-webhook-id'];

  // Check if already processed (idempotency)
  if (processedWebhooks.has(webhookId)) {
    console.log(`Webhook ${webhookId} already processed, skipping`);
    return res.status(200).json({ received: true });
  }

  try {
    // Process webhook
    await processWebhook(req.body);

    // Mark as processed
    processedWebhooks.add(webhookId);

    // Respond success
    res.status(200).json({ received: true });

  } catch (err) {
    console.error('Webhook processing failed:', err);

    // Return 500 to trigger retry
    res.status(500).json({ error: 'Processing failed' });
  }
});

// Clean up old webhook IDs periodically
setInterval(() => {
  processedWebhooks.clear();
}, 24 * 60 * 60 * 1000); // Clear daily

Better approach with database:

app.post('/webhooks/endpoint', async (req, res) => {
  const webhookId = req.headers['x-webhook-id'];

  try {
    // Check if webhook already processed
    const existing = await db.webhooks.findOne({ webhookId });

    if (existing) {
      console.log(`Duplicate webhook ${webhookId}, already processed`);
      return res.status(200).json({ received: true });
    }

    // Store webhook receipt
    await db.webhooks.insert({
      webhookId,
      payload: req.body,
      receivedAt: new Date(),
      status: 'processing'
    });

    // Respond quickly
    res.status(200).json({ received: true });

    // Process asynchronously
    await queue.add('process-webhook', {
      webhookId,
      payload: req.body
    });

  } catch (err) {
    console.error('Error handling webhook:', err);
    res.status(500).json({ error: 'Internal server error' });
  }
});

Fix 4: Handle Webhook Retries

Understand and configure retry behavior:

Common retry patterns:

• Immediate retry
• Exponential backoff (1min, 5min, 15min, 1hr, etc.)
• Maximum retry attempts (usually 3-10)
• Manual retry option

Best practices:

// Respond with appropriate status codes

// Success - no retry needed
res.status(200).send('OK');

// Temporary error - will retry
res.status(500).send('Temporary error');
res.status(502).send('Bad gateway');
res.status(503).send('Service unavailable');
res.status(504).send('Gateway timeout');

// Permanent error - will NOT retry
res.status(400).send('Bad request');
res.status(401).send('Unauthorized');
res.status(403).send('Forbidden');
res.status(404).send('Not found');

Configure retry settings in webhook provider:

// Example: Configure Stripe webhook with retry settings
// (Done in Stripe dashboard, not code)

// Retry schedule:
// - Immediately
// - 5 minutes
// - 30 minutes
// - 2 hours
// - 12 hours
// - After 12 hours, retries end

// Maximum attempts: 5-10 (varies by provider)

Fix 5: Implement Webhook Queue Processing

Process webhooks asynchronously for reliability:

const Queue = require('bull');
const webhookQueue = new Queue('webhooks', {
  redis: {
    host: 'localhost',
    port: 6379
  }
});

// Webhook endpoint - receives and queues
app.post('/webhooks/endpoint', async (req, res) => {
  const webhookId = req.headers['x-webhook-id'];

  try {
    // Add to queue immediately
    await webhookQueue.add({
      id: webhookId,
      type: req.body.type,
      payload: req.body,
      receivedAt: Date.now()
    }, {
      attempts: 3,
      backoff: {
        type: 'exponential',
        delay: 2000
      }
    });

    // Respond quickly
    res.status(200).json({ received: true });

  } catch (err) {
    console.error('Failed to queue webhook:', err);
    res.status(500).json({ error: 'Failed to queue' });
  }
});

// Queue processor - handles actual processing
webhookQueue.process(async (job) => {
  const { id, type, payload } = job.data;

  console.log(`Processing webhook ${id} (${type})`);

  try {
    // Process webhook based on type
    switch (type) {
      case 'order.created':
        await handleOrderCreated(payload);
        break;
      case 'payment.succeeded':
        await handlePaymentSucceeded(payload);
        break;
      default:
        console.log(`Unknown webhook type: ${type}`);
    }

    console.log(`Webhook ${id} processed successfully`);

  } catch (err) {
    console.error(`Error processing webhook ${id}:`, err);
    throw err; // Will retry
  }
});

// Monitor queue
webhookQueue.on('failed', (job, err) => {
  console.error(`Webhook job ${job.id} failed:`, err.message);
  // Alert on repeated failures
});

webhookQueue.on('completed', (job) => {
  console.log(`Webhook job ${job.id} completed`);
});

Fix 6: Validate Webhook Payload

Validate payload structure before processing:

const Joi = require('joi');

// Define expected payload schema
const orderWebhookSchema = Joi.object({
  type: Joi.string().valid('order.created').required(),
  data: Joi.object({
    id: Joi.string().required(),
    amount: Joi.number().positive().required(),
    currency: Joi.string().length(3).required(),
    customer: Joi.object({
      email: Joi.string().email().required(),
      name: Joi.string().required()
    }).required()
  }).required(),
  created_at: Joi.date().iso().required()
});

app.post('/webhooks/orders', async (req, res) => {
  try {
    // Validate payload
    const { error, value } = orderWebhookSchema.validate(req.body);

    if (error) {
      console.error('Invalid webhook payload:', error.details);
      return res.status(400).json({
        error: 'Invalid payload',
        details: error.details
      });
    }

    // Payload valid, process it
    await processOrderWebhook(value);
    res.status(200).json({ received: true });

  } catch (err) {
    console.error('Webhook processing error:', err);
    res.status(500).json({ error: 'Processing failed' });
  }
});

Platform-Specific Guides

Detailed webhook implementation for your specific platform:

Verification

After implementing fixes:

1. Test webhook delivery:

   • Trigger test event in provider dashboard
   • Verify webhook received
   • Check logs for receipt
   • Confirm processing completed
   • Verify data updated correctly

2. Test signature verification:

   • Valid signature accepted
   • Invalid signature rejected (400)
   • Missing signature rejected
   • Check logs for verification attempts

3. Test retry behavior:

   • Return 500 to trigger retry
   • Verify retries happen
   • Check retry timing
   • Confirm eventual success
   • Test retry exhaustion

4. Test idempotency:

   • Send duplicate webhooks
   • Verify only processed once
   • No duplicate data created
   • Check deduplication logs

5. Load testing:

   • Send multiple webhooks rapidly
   • Verify all processed
   • Check for dropped webhooks
   • Monitor queue depth
   • Test under high load

Common Mistakes

1. Slow response time - Respond within 10 seconds
2. No signature verification - Always verify authenticity
3. Synchronous processing - Use queues for heavy work
4. No idempotency - Handle duplicate deliveries
5. Wrong status codes - Return appropriate codes
6. Invalid SSL certificate - Keep certificates current
7. No error logging - Log all webhook failures
8. Firewall blocking - Whitelist webhook IPs
9. No retry handling - Expect and handle retries
10. Missing payload validation - Validate structure

Troubleshooting Checklist

• Endpoint URL publicly accessible
• SSL certificate valid and current
• Webhook signature verification working
• Responding within 10 seconds
• Appropriate HTTP status codes
• Idempotency implemented
• Async processing for heavy work
• Error logging enabled
• Retry logic tested
• Payload validation implemented
• Queue monitoring set up
• Firewall rules configured
• Test webhooks working

Further Reading

• Webhook Best Practices
• Stripe Webhooks Guide
• Shopify Webhooks Documentation
• Building Reliable Webhooks
• Integration Issues Overview
• API Authentication