Webhook Development Complete Guide: Architecture, Security, and Best Practices

┌─────────────────────────────────────────────────────────────────────┐
│              Webhook vs Polling Architecture                         │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  POLLING (Traditional)                                               │
│  ═════════════════════                                               │
│                                                                      │
│  Your Server              Provider API                               │
│      │                        │                                      │
│      │──── GET /events? ─────▶│                                      │
│      │◀─── No new events ─────│                                      │
│      │                        │                                      │
│      │──── GET /events? ─────▶│  (repeat every N seconds)           │
│      │◀─── No new events ─────│                                      │
│      │                        │                                      │
│      │──── GET /events? ─────▶│                                      │
│      │◀─── [Event data!] ─────│  (finally something)                │
│      │                        │                                      │
│                                                                      │
│  Problems: Wasted requests, delayed detection, server load          │
│                                                                      │
│  ═══════════════════════════════════════════════════════════════    │
│                                                                      │
│  WEBHOOKS (Event-Driven)                                             │
│  ═══════════════════════                                             │
│                                                                      │
│  Your Server              Provider System                            │
│      │                        │                                      │
│      │                        │──── Event occurs!                    │
│      │◀── POST /webhook ──────│                                      │
│      │─── 200 OK ────────────▶│                                      │
│      │                        │                                      │
│      │                        │──── Another event!                   │
│      │◀── POST /webhook ──────│                                      │
│      │─── 200 OK ────────────▶│                                      │
│      │                        │                                      │
│                                                                      │
│  Benefits: Real-time, efficient, scalable, lower latency            │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────┐
│              Webhook System Components                               │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                    WEBHOOK PROVIDER                           │   │
│  │                                                               │   │
│  │  ┌─────────────┐   ┌─────────────┐   ┌─────────────────┐    │   │
│  │  │   Event     │──▶│  Delivery   │──▶│  HTTP Client    │    │   │
│  │  │   Source    │   │   Queue     │   │  (with retries) │    │   │
│  │  └─────────────┘   └─────────────┘   └────────┬────────┘    │   │
│  │                                               │              │   │
│  └───────────────────────────────────────────────┼──────────────┘   │
│                                                  │                   │
│                     ┌────────────────────────────┘                  │
│                     │ HTTPS POST + Signature                        │
│                     ▼                                               │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │                    WEBHOOK CONSUMER                           │   │
│  │                                                               │   │
│  │  ┌─────────────┐   ┌─────────────┐   ┌─────────────────┐    │   │
│  │  │  Endpoint   │──▶│  Signature  │──▶│   Event Queue   │    │   │
│  │  │  Handler    │   │  Validator  │   │  (async proc)   │    │   │
│  │  └─────────────┘   └─────────────┘   └────────┬────────┘    │   │
│  │         │                                      │              │   │
│  │         │ Return 200                          │              │   │
│  │         ▼                                      ▼              │   │
│  │  ┌─────────────┐                    ┌─────────────────┐     │   │
│  │  │  Response   │                    │  Background     │     │   │
│  │  │  (fast!)    │                    │  Workers        │     │   │
│  │  └─────────────┘                    └─────────────────┘     │   │
│  │                                                               │   │
│  └──────────────────────────────────────────────────────────────┘   │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

// Express.js webhook handler with best practices
import express from 'express';
import crypto from 'crypto';

const app = express();

// Use raw body for signature verification
app.use('/webhooks', express.raw({ type: 'application/json' }));

// Webhook endpoint
app.post('/webhooks/stripe', async (req, res) => {
  const startTime = Date.now();
  const requestId = crypto.randomUUID();

  try {
    // 1. Verify signature FIRST
    const signature = req.headers['stripe-signature'];
    const webhookSecret = process.env.STRIPE_WEBHOOK_SECRET!;

    const event = verifyStripeSignature(req.body, signature, webhookSecret);

    // 2. Check for duplicates (idempotency)
    if (await isDuplicateEvent(event.id)) {
      console.log(`Duplicate event ${event.id}, skipping`);
      return res.status(200).json({ received: true, duplicate: true });
    }

    // 3. Mark event as received
    await markEventReceived(event.id);

    // 4. Queue for async processing
    await eventQueue.add('stripe-webhook', {
      eventId: event.id,
      eventType: event.type,
      payload: event,
      receivedAt: new Date().toISOString(),
      requestId
    });

    // 5. Log and respond quickly
    console.log({
      requestId,
      eventId: event.id,
      eventType: event.type,
      processingTime: Date.now() - startTime
    });

    return res.status(200).json({ received: true });

  } catch (error) {
    if (error.message === 'Invalid signature') {
      console.error({ requestId, error: 'Invalid signature', ip: req.ip });
      return res.status(401).json({ error: 'Invalid signature' });
    }

    console.error({ requestId, error: error.message });
    return res.status(500).json({ error: 'Internal error' });
  }
});

// Signature verification
function verifyStripeSignature(
  payload: Buffer,
  signatureHeader: string,
  secret: string
): any {
  const parts = signatureHeader.split(',');
  const timestamp = parts.find(p => p.startsWith('t='))?.split('=')[1];
  const signature = parts.find(p => p.startsWith('v1='))?.split('=')[1];

  if (!timestamp || !signature) {
    throw new Error('Invalid signature');
  }

  // Prevent replay attacks - reject old events
  const currentTime = Math.floor(Date.now() / 1000);
  if (currentTime - parseInt(timestamp) > 300) { // 5 minutes
    throw new Error('Invalid signature'); // Timestamp too old
  }

  // Compute expected signature
  const signedPayload = `${timestamp}.${payload.toString()}`;
  const expectedSignature = crypto
    .createHmac('sha256', secret)
    .update(signedPayload)
    .digest('hex');

  // Timing-safe comparison
  if (!crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expectedSignature)
  )) {
    throw new Error('Invalid signature');
  }

  return JSON.parse(payload.toString());
}

// Idempotency check
async function isDuplicateEvent(eventId: string): Promise<boolean> {
  // Check Redis/database for existing event
  const exists = await redis.exists(`webhook:event:${eventId}`);
  return exists === 1;
}

async function markEventReceived(eventId: string): Promise<void> {
  // Store with 7-day TTL to handle late retries
  await redis.setex(`webhook:event:${eventId}`, 7 * 24 * 60 * 60, 'received');
}

// Worker that processes queued webhook events
import { Worker } from 'bullmq';

const worker = new Worker('stripe-webhook', async (job) => {
  const { eventId, eventType, payload, requestId } = job.data;

  console.log({
    worker: 'stripe-webhook',
    eventId,
    eventType,
    requestId,
    attempt: job.attemptsMade + 1
  });

  try {
    // Route to appropriate handler based on event type
    switch (eventType) {
      case 'payment_intent.succeeded':
        await handlePaymentSuccess(payload);
        break;

      case 'payment_intent.payment_failed':
        await handlePaymentFailure(payload);
        break;

      case 'customer.subscription.created':
        await handleSubscriptionCreated(payload);
        break;

      case 'customer.subscription.deleted':
        await handleSubscriptionCanceled(payload);
        break;

      case 'invoice.paid':
        await handleInvoicePaid(payload);
        break;

      case 'invoice.payment_failed':
        await handleInvoicePaymentFailed(payload);
        break;

      default:
        console.log({ eventType, message: 'Unhandled event type' });
    }

    // Mark as fully processed
    await markEventProcessed(eventId);

  } catch (error) {
    console.error({
      eventId,
      eventType,
      error: error.message,
      attempt: job.attemptsMade + 1
    });
    throw error; // Triggers retry
  }
}, {
  connection: redisConnection,
  concurrency: 10,  // Process 10 events concurrently

  // Retry configuration
  settings: {
    backoffStrategy: (attemptsMade) => {
      // Exponential backoff: 1m, 5m, 15m, 30m, 1h
      const delays = [60000, 300000, 900000, 1800000, 3600000];
      return delays[Math.min(attemptsMade, delays.length - 1)];
    }
  }
});

// Handle permanently failed jobs
worker.on('failed', async (job, error) => {
  if (job.attemptsMade >= 5) {
    // Move to dead letter queue
    await deadLetterQueue.add('failed-webhook', {
      originalJob: job.data,
      error: error.message,
      attempts: job.attemptsMade,
      failedAt: new Date().toISOString()
    });

    // Alert operations team
    await alerting.send({
      severity: 'high',
      message: `Webhook permanently failed after ${job.attemptsMade} attempts`,
      eventId: job.data.eventId,
      eventType: job.data.eventType,
      error: error.message
    });
  }
});

// Handlers implement idempotent operations
async function handlePaymentSuccess(event: any): Promise<void> {
  const paymentIntent = event.data.object;
  const orderId = paymentIntent.metadata.order_id;

  // Idempotent: only update if not already processed
  const order = await db.order.findUnique({ where: { id: orderId } });

  if (order.status === 'paid') {
    console.log({ orderId, message: 'Order already paid, skipping' });
    return;
  }

  await db.$transaction([
    db.order.update({
      where: { id: orderId },
      data: {
        status: 'paid',
        paidAt: new Date(),
        paymentIntentId: paymentIntent.id
      }
    }),
    db.payment.create({
      data: {
        orderId,
        amount: paymentIntent.amount,
        currency: paymentIntent.currency,
        paymentIntentId: paymentIntent.id
      }
    })
  ]);

  // Trigger downstream actions
  await emailService.sendOrderConfirmation(order);
  await inventoryService.reserveItems(order);
}

┌─────────────────────────────────────────────────────────────────────┐
│              Webhook Security Layers                                 │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  Layer 1: TRANSPORT SECURITY                                         │
│  ════════════════════════════                                        │
│  • HTTPS only (TLS 1.2+)                                            │
│  • Valid SSL certificate                                            │
│  • No self-signed certs in production                               │
│                                                                      │
│  Layer 2: AUTHENTICATION (Signature Verification)                    │
│  ═══════════════════════════════════════════════                     │
│  • HMAC-SHA256 signature in header                                  │
│  • Shared secret between provider and consumer                      │
│  • Timing-safe comparison to prevent timing attacks                 │
│                                                                      │
│  Layer 3: REPLAY PROTECTION                                          │
│  ══════════════════════════                                          │
│  • Timestamp in signature (reject >5 min old)                       │
│  • Idempotency keys (track processed events)                        │
│  • Nonce validation (if supported)                                  │
│                                                                      │
│  Layer 4: NETWORK CONTROLS (Optional)                                │
│  ════════════════════════════════════                                │
│  • IP allowlisting (if provider publishes IPs)                      │
│  • VPN/private endpoints for sensitive data                         │
│  • Rate limiting on webhook endpoint                                │
│                                                                      │
│  Layer 5: DATA VALIDATION                                            │
│  ═════════════════════════                                           │
│  • Schema validation of payload                                     │
│  • Business logic validation                                        │
│  • Sanitize before use (treat as untrusted input)                  │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

// Provider-specific signature verification

// Stripe: HMAC-SHA256 with timestamp
function verifyStripeSignature(payload: string, header: string, secret: string): boolean {
  const elements = header.split(',');
  const timestamp = elements.find(e => e.startsWith('t='))?.slice(2);
  const signature = elements.find(e => e.startsWith('v1='))?.slice(3);

  const signedPayload = `${timestamp}.${payload}`;
  const expected = crypto.createHmac('sha256', secret).update(signedPayload).digest('hex');

  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}

// GitHub: HMAC-SHA256
function verifyGitHubSignature(payload: string, header: string, secret: string): boolean {
  const signature = header.replace('sha256=', '');
  const expected = crypto.createHmac('sha256', secret).update(payload).digest('hex');

  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}

// Slack: HMAC-SHA256 with timestamp
function verifySlackSignature(
  payload: string,
  signature: string,
  timestamp: string,
  secret: string
): boolean {
  const baseString = `v0:${timestamp}:${payload}`;
  const expected = 'v0=' + crypto.createHmac('sha256', secret).update(baseString).digest('hex');

  return crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected));
}

// Shopify: HMAC-SHA256 base64
function verifyShopifySignature(payload: string, header: string, secret: string): boolean {
  const expected = crypto.createHmac('sha256', secret).update(payload).digest('base64');

  return crypto.timingSafeEqual(Buffer.from(header), Buffer.from(expected));
}

// Generic: HMAC with configurable algorithm
function verifyHMACSignature(
  payload: string,
  signature: string,
  secret: string,
  algorithm: 'sha256' | 'sha1' = 'sha256',
  encoding: 'hex' | 'base64' = 'hex'
): boolean {
  const expected = crypto
    .createHmac(algorithm, secret)
    .update(payload)
    .digest(encoding);

  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expected)
  );
}

// Event handler registry pattern
type WebhookHandler = (event: WebhookEvent) => Promise<void>;

interface HandlerRegistry {
  [eventType: string]: WebhookHandler;
}

class WebhookProcessor {
  private handlers: HandlerRegistry = {};
  private defaultHandler?: WebhookHandler;

  // Register handlers for specific event types
  on(eventType: string, handler: WebhookHandler): this {
    this.handlers[eventType] = handler;
    return this;
  }

  // Register handler for unhandled events
  onDefault(handler: WebhookHandler): this {
    this.defaultHandler = handler;
    return this;
  }

  // Process an event
  async process(event: WebhookEvent): Promise<void> {
    const handler = this.handlers[event.type] || this.defaultHandler;

    if (!handler) {
      console.log({ eventType: event.type, message: 'No handler registered' });
      return;
    }

    await handler(event);
  }
}

// Usage: Build a processor for Stripe webhooks
const stripeProcessor = new WebhookProcessor()
  // Payment events
  .on('payment_intent.succeeded', handlePaymentSuccess)
  .on('payment_intent.payment_failed', handlePaymentFailed)
  .on('charge.refunded', handleRefund)
  .on('charge.dispute.created', handleDispute)

  // Subscription events
  .on('customer.subscription.created', handleSubscriptionCreated)
  .on('customer.subscription.updated', handleSubscriptionUpdated)
  .on('customer.subscription.deleted', handleSubscriptionDeleted)
  .on('customer.subscription.trial_will_end', handleTrialEnding)

  // Invoice events
  .on('invoice.paid', handleInvoicePaid)
  .on('invoice.payment_failed', handleInvoiceFailed)
  .on('invoice.upcoming', handleUpcomingInvoice)

  // Customer events
  .on('customer.created', handleCustomerCreated)
  .on('customer.updated', handleCustomerUpdated)
  .on('customer.deleted', handleCustomerDeleted)

  // Default handler for unregistered events
  .onDefault(async (event) => {
    console.log({ eventType: event.type, message: 'Unhandled event type' });
    // Optionally store for later analysis
    await storeUnhandledEvent(event);
  });

// In your worker
worker.on('stripe-webhook', async (job) => {
  await stripeProcessor.process(job.data.payload);
});

// Webhook metrics and monitoring
import { Counter, Histogram, Gauge } from 'prom-client';

// Metrics
const webhooksReceived = new Counter({
  name: 'webhooks_received_total',
  help: 'Total webhooks received',
  labelNames: ['provider', 'event_type', 'status']
});

const webhookProcessingDuration = new Histogram({
  name: 'webhook_processing_duration_seconds',
  help: 'Webhook processing duration',
  labelNames: ['provider', 'event_type'],
  buckets: [0.1, 0.5, 1, 2, 5, 10, 30]
});

const webhookQueueDepth = new Gauge({
  name: 'webhook_queue_depth',
  help: 'Current webhook queue depth',
  labelNames: ['provider']
});

const webhookRetries = new Counter({
  name: 'webhook_retries_total',
  help: 'Total webhook processing retries',
  labelNames: ['provider', 'event_type']
});

// Instrumented handler
async function handleWebhook(req: Request, res: Response) {
  const startTime = Date.now();
  const provider = req.params.provider;
  let eventType = 'unknown';
  let status = 'success';

  try {
    const event = await validateAndParse(req);
    eventType = event.type;

    await processWebhook(event);

  } catch (error) {
    status = error.message === 'Invalid signature' ? 'invalid_signature' : 'error';
    throw error;
  } finally {
    // Record metrics
    webhooksReceived.inc({ provider, event_type: eventType, status });
    webhookProcessingDuration.observe(
      { provider, event_type: eventType },
      (Date.now() - startTime) / 1000
    );
  }
}

// Dashboard queries (Prometheus/Grafana)
const dashboardQueries = {
  // Success rate
  successRate: `
    sum(rate(webhooks_received_total{status="success"}[5m]))
    / sum(rate(webhooks_received_total[5m])) * 100
  `,

  // Average processing time
  avgProcessingTime: `
    histogram_quantile(0.95,
      rate(webhook_processing_duration_seconds_bucket[5m])
    )
  `,

  // Error rate by type
  errorsByType: `
    sum by (event_type) (
      rate(webhooks_received_total{status!="success"}[5m])
    )
  `,

  // Queue depth
  queueDepth: `webhook_queue_depth`,

  // Retry rate
  retryRate: `
    sum(rate(webhook_retries_total[5m]))
    / sum(rate(webhooks_received_total[5m])) * 100
  `
};

// Alerting rules
const alertRules = `
groups:
  - name: webhook_alerts
    rules:
      - alert: WebhookHighErrorRate
        expr: |
          sum(rate(webhooks_received_total{status!="success"}[5m]))
          / sum(rate(webhooks_received_total[5m])) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: Webhook error rate above 5%

      - alert: WebhookQueueBacklog
        expr: webhook_queue_depth > 1000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Webhook queue depth exceeding 1000

      - alert: WebhookSlowProcessing
        expr: |
          histogram_quantile(0.95,
            rate(webhook_processing_duration_seconds_bucket[5m])
          ) > 30
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: Webhook p95 processing time exceeding 30s

      - alert: WebhookSignatureFailures
        expr: rate(webhooks_received_total{status="invalid_signature"}[5m]) > 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: High rate of webhook signature failures
`;

┌─────────────────────────────────────────────────────────────────────┐
│              Webhook Development Learning Path                       │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│  BEGINNER                                                           │
│  ════════                                                           │
│  1. Webhooks Explained - Core concepts and terminology              │
│  2. Webhook Best Practices - Essential patterns                     │
│  3. Set up a simple webhook endpoint                                │
│  4. Use ngrok for local testing                                     │
│                                                                      │
│  INTERMEDIATE                                                        │
│  ════════════                                                        │
│  5. Signature Verification Guide - Implement security               │
│  6. Retry Logic Guide - Handle failures gracefully                  │
│  7. Testing & Debugging Guide - Build test harnesses                │
│  8. Implement async processing with queues                          │
│                                                                      │
│  ADVANCED                                                           │
│  ════════                                                           │
│  9. Scaling & Performance Guide - Handle high volume                │
│  10. Error Handling Guide - Dead letter queues, recovery            │
│  11. Platform Integrations - Provider-specific patterns             │
│  12. Build comprehensive monitoring                                 │
│                                                                      │
│  EXPERT                                                             │
│  ══════                                                             │
│  13. Building Webhook Providers - Design your own system            │
│  14. Multi-tenant webhook infrastructure                            │
│  15. Webhook security hardening                                     │
│  16. Distributed webhook processing                                 │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘

□ HTTPS only (valid certificate)
□ Signature verification implemented
□ Timestamp validation (reject old requests)
□ Idempotency handling (track event IDs)
□ Async processing (queue events)
□ Fast response (<5 seconds)
□ Return 200 on success
□ Return 401 on invalid signature
□ Return 5xx only for retryable errors
□ Comprehensive logging
□ Monitoring and alerting
□ Dead letter queue for failures

Received → Validated → Queued → Processing → Completed
                                    ↓
                                 Failed → Retry (up to N times)
                                    ↓
                              Dead Letter Queue