WooCommerce Webhook Delivery Failures: Building a Circuit Breaker & Exponential Backoff for woocommerce_deliver_webhook_asyn

Illustration of an outbound WooCommerce webhook pipeline protected by a host-level circuit breaker and an Action Scheduler exponential backoff retry queue.

Introduction: What an ERP Outage Does to a WooCommerce Queue

WooCommerce does not stop producing events when your ERP, 3PL, or middleware endpoint goes down. Every order creation, status transition, and refund still queues a woocommerce_deliver_webhook_async action in Action Scheduler, and each action makes a blocking outbound HTTP request. When the receiver is unhealthy, that design produces two separate failures, and both carry commercial cost:

  1. Action Scheduler worker starvation. Deliveries to a hanging endpoint hold Action Scheduler background queue runners for the full length of the HTTP timeout. Payment-related and subscription-renewal actions sit behind them in the same queue, so a fulfillment integration problem becomes a checkout and billing problem. Under sustained traffic, blocking outbound HTTP requests can exhaust available queue runner concurrency, increase background job latency, and create cascading failures across unrelated WooCommerce scheduled actions.
  2. Silent auto-disable. After the failure count passes the core threshold (5 by default), WooCommerce flips the webhook from active to disabled. New order and fulfillment events stop being queued. Nothing pages anyone, and the warehouse simply stops receiving orders until someone notices.

For a store with a 3PL SLA or a same-day-ship commitment, either outcome is an incident. This article covers what core actually does on a failed delivery, how to diagnose the damage with SQL and WP-CLI, and a must-use plugin that implements a woocommerce_deliver_webhook_async backlog retry circuit breaker with jittered exponential backoff, so an outage degrades into a delayed delivery instead of lost events.

Two Myths: Longer Timeouts and “Built-In” Exponential Backoff

Troubleshooting advice for this problem, particularly generated advice, tends to repeat two claims. Both are wrong, and both make production worse.

Myth 1: Raise the timeout to 30–60 seconds in woocommerce_webhook_http_args

The reasoning sounds plausible: a slow ERP needs more time to respond. In practice, a 60-second timeout means that when the endpoint hangs, every woocommerce_deliver_webhook_async worker blocks for a full minute. Action Scheduler runs a finite number of concurrent runners, and each batch has a time limit (30 seconds by default). A single webhook call that can outlast the batch limit occupies a concurrency slot while every other pending action waits. The backlog grows faster than it drains, and the delay spreads to unrelated hooks on the same queue. The same applies to WP-CLI and cron-triggered runners, which are tied up for the duration of the hang.

The correct posture is the opposite: a short, strict timeout (5 seconds is a reasonable ceiling for a webhook receiver that should acknowledge and queue), combined with a retry mechanism that does the waiting outside the worker.

Myth 2: WooCommerce retries failed webhooks with exponential backoff

It does not. Core (includes/class-wc-webhook.php) has no exponential backoff for 5xx or timeout responses. This is what WC_Webhook::deliver() does when the response is not a success:

  1. It sends the request and inspects the HTTP response code.
  2. If the code falls outside the 200–299 range (a WP_Error yields no code at all), it calls WC_Webhook::failed_delivery().
  3. failed_delivery() increments the failure counter on the webhook record. Once the count exceeds the value returned by the woocommerce_max_webhook_delivery_failures filter (default 5), the webhook status is changed from active to disabled.
  4. deliver() returns normally. No PHP exception is thrown, so Action Scheduler records the woocommerce_deliver_webhook_async action as complete, not failed—just as wp_mail() swallows transport failures unless you capture SMTP errors and schedule bounded Action Scheduler retries.

The consequence is that a failed delivery looks like a successful job in the Action Scheduler admin screen. There is no retry, no failed-action entry to alert on, and after the sixth consecutive failure the webhook stops firing entirely. Order data is not queued for later; it is skipped.

The opposite mistake: throwing an exception on every failure

Once developers learn that failures are swallowed, the next idea is to throw an exception from a hook so Action Scheduler marks the action failed and retries. Without a circuit breaker this is just as damaging. During a 30-minute ERP outage on a flash-sale day, every queued delivery fails, every failure produces a new immediate retry, and wp_actionscheduler_actions fills with tens of thousands of pending rows that all hit the same dead endpoint. You have replaced a silent failure with a self-inflicted denial of service against your own queue and your own receiver. In distributed systems, this synchronized retry behavior is known as the thundering herd problem. When thousands of failed WooCommerce webhook deliveries attempt to reconnect to the same ERP endpoint simultaneously, the resulting retry storm can overwhelm both the recovering service and the originating application, potentially triggering another outage.

The fix needs three properties together: a hard timeout, a host-level circuit breaker that fails fast while the endpoint is down, and retries scheduled with exponential delay plus jitter so recovery does not produce a thundering herd.

Background references: legacy webhook schema upgrades live in wc_update_300_webhooks() (includes/wc-update-functions.php), asynchronous dispatch is defined in includes/wc-webhook-functions.php, and queue-starvation and delivery regressions are discussed in WooCommerce GitHub issue #48741.

Flowchart comparing default WC_Webhook 60-second hangs and 5-failure auto-disables against a 5-second timeout, pre_http_request circuit breaker, and jittered backoff retry queue.

Diagnostic Workflow

Start by establishing which webhooks have been silently disabled and how large the woocommerce_deliver_webhook_async backlog is. Replace the wp_ table prefix if your install uses a different one.

Query 1: webhook health and silent auto-disabling

-- 1. Inspect all registered webhooks for failure counts and silent auto-disabling
SELECT
    webhook_id,
    name,
    status,
    delivery_url,
    topic,
    failure_count,
    date_modified_gmt
FROM wp_wc_webhooks
ORDER BY failure_count DESC, status ASC;

Look for rows with status = 'disabled' that you did not disable, and rows where failure_count is climbing toward the threshold. date_modified_gmt on a disabled row usually tells you when the integration went dark.

Query 2: backlog volume and age

-- 2. Check backlog volume and execution age of woocommerce_deliver_webhook_async actions
SELECT
    status,
    COUNT(action_id)       AS job_count,
    MIN(scheduled_date_gmt) AS oldest_job_gmt,
    MAX(scheduled_date_gmt) AS newest_job_gmt,
    TIMESTAMPDIFF(MINUTE, MIN(scheduled_date_gmt), UTC_TIMESTAMP()) AS oldest_age_minutes
FROM wp_actionscheduler_actions
WHERE hook = 'woocommerce_deliver_webhook_async'
GROUP BY status;

A large pending count with an old oldest_job_gmt means workers are not draining the queue, which is the starvation pattern. A high complete count alongside a high failure_count in Query 1 confirms the swallowed-failure pattern described above: the jobs finished, the deliveries did not succeed.

WP-CLI: failure counts per webhook

# 3. Inspect webhook status and failure counts via WP-CLI
wp eval 'foreach ( WC_Data_Store::load( "webhook" )->get_webhooks_ids() as $id ) { $wh = wc_get_webhook( $id ); echo "{$id}: {$wh->get_name()} | Status: {$wh->get_status()} | Failures: {$wh->get_failure_count()}\n"; }' --allow-root

The --allow-root flag is only needed when WP-CLI runs as root; drop it otherwise. Loading IDs through WC_Data_Store::load( "webhook" )->get_webhooks_ids() queries the core webhook data store directly (note that the data store caches the ID list in a transient).

To count the pending backlog from the shell without SQL:

wp action-scheduler action list --hook=woocommerce_deliver_webhook_async --status=pending --format=count

Production-Ready Solution: A Must-Use Plugin

The plugin below is a stateful implementation of the woocommerce_deliver_webhook_async backlog retry circuit breaker. It does five things:

  • Raises woocommerce_max_webhook_delivery_failures to 10000, so transient ERP maintenance never disables a webhook.
  • Enforces a 5-second timeout in woocommerce_webhook_http_args and initializes an X-WC-Webhook-Attempt header.
  • Runs a pre-flight, host-level circuit breaker on pre_http_request. If the target host has recorded 5 or more failures inside the 120-second window, the outbound call is skipped and a WP_Error('wc_webhook_circuit_open', ...) is returned immediately, so workers fail in under a millisecond instead of waiting on a dead socket.
  • Inspects every delivery on woocommerce_webhook_delivery: resets the breaker on 2xx, ignores permanent 4xx errors (except 408 and 429), and on 5xx, 408, 429, or WP_Error increments the breaker and schedules a retry.
  • Schedules retries in a dedicated wc-webhooks-retry Action Scheduler group with delay 2 ** $attempt * 30 + random_int(0, 15) seconds, up to 6 total attempts.

Save it as wp-content/mu-plugins/enterprise-wc-webhook-resilience.php. Must-use plugins load before WooCommerce, which is fine here because every WooCommerce and Action Scheduler call happens inside a callback at runtime.

<?php
/**
 * Plugin Name: Enterprise WooCommerce Webhook Circuit Breaker & Exponential Backoff
 * Description: Prevents ERP outages from starving Action Scheduler or auto-disabling webhooks; adds jittered backoff.
 * Version: 1.0.0
 */

declare(strict_types=1);

namespace EnterpriseWC\WebhookResilience;

use WC_Webhook;
use WP_Error;

if ( ! defined( 'ABSPATH' ) ) {
    exit;
}

final class WebhookCircuitBreaker {

    private const RETRY_HOOK        = 'enterprise_wc_webhook_exponential_retry';
    private const RETRY_GROUP       = 'wc-webhooks-retry';
    private const MAX_ATTEMPTS      = 6;
    private const CIRCUIT_THRESHOLD = 5;
    private const CIRCUIT_COOLDOWN  = 120; // Seconds: circuit-open / failure-counting window.
    private const OPEN_ERROR_CODE   = 'wc_webhook_circuit_open';

    /**
     * Register webhook delivery resilience filters and the retry worker.
     */
    public static function init(): void {
        // 1. Stop WooCommerce from silently disabling webhooks after 5 transient failures.
        add_filter( 'woocommerce_max_webhook_delivery_failures', [ self::class, 'set_high_failure_threshold' ] );

        // 2. Strict 5-second timeout so slow endpoints cannot block workers.
        add_filter( 'woocommerce_webhook_http_args', [ self::class, 'enforce_strict_http_timeouts' ], 10, 3 );

        // 3. Pre-flight: if the circuit is OPEN for the target host, fail immediately.
        add_filter( 'pre_http_request', [ self::class, 'short_circuit_open_endpoints' ], 10, 3 );

        // 4. Inspect each delivery; manage circuit state; schedule backoff retries.
        add_action( 'woocommerce_webhook_delivery', [ self::class, 'handle_delivery_outcome' ], 10, 5 );

        // 5. Dedicated retry worker.
        add_action( self::RETRY_HOOK, [ self::class, 'execute_backoff_retry' ], 10, 3 );
    }

    /**
     * Raise the auto-disable threshold from 5 to 10,000.
     */
    public static function set_high_failure_threshold(): int {
        return 10000;
    }

    /**
     * Cap the HTTP timeout at 5 seconds and initialize the attempt header.
     *
     * @param array $http_args  wp_remote_request arguments.
     * @param mixed $arg        Webhook payload argument.
     * @param int   $webhook_id Webhook ID.
     */
    public static function enforce_strict_http_timeouts( array $http_args, $arg, int $webhook_id ): array {
        $http_args['timeout'] = 5;
        $http_args['headers']['X-WC-Webhook-Attempt'] = (string) ( $http_args['headers']['X-WC-Webhook-Attempt'] ?? 1 );

        return $http_args;
    }

    /**
     * Deterministic transient key per target host.
     */
    private static function get_circuit_key( string $url ): string {
        $host = (string) wp_parse_url( $url, PHP_URL_HOST );

        return 'wc_wh_cb_' . md5( $host !== '' ? $host : $url );
    }

    /**
     * Skip the outbound request when the circuit for the target host is OPEN.
     *
     * @param false|array|WP_Error $preempt     Whether to preempt the HTTP request.
     * @param array                $parsed_args HTTP request arguments.
     * @param string               $url         Target URL.
     * @return false|array|WP_Error
     */
    public static function short_circuit_open_endpoints( $preempt, $parsed_args, $url ) {
        // Only act on WooCommerce webhook deliveries.
        if ( ! is_array( $parsed_args ) || empty( $parsed_args['headers']['X-WC-Webhook-ID'] ) ) {
            return $preempt;
        }

        $failures = (int) get_transient( self::get_circuit_key( (string) $url ) );

        if ( $failures >= self::CIRCUIT_THRESHOLD ) {
            return new WP_Error(
                self::OPEN_ERROR_CODE,
                sprintf( 'Circuit breaker OPEN for endpoint %s (%d recent failures). Fast-failing to retry queue.', $url, $failures )
            );
        }

        return $preempt;
    }

    /**
     * Evaluate the delivery result, update circuit state, and schedule a jittered retry.
     *
     * @param array          $http_args  HTTP request arguments.
     * @param array|WP_Error $response   HTTP response or WP_Error.
     * @param mixed          $duration   Request duration in seconds.
     * @param mixed          $arg        First webhook argument (for example, an order ID).
     * @param int            $webhook_id Webhook ID.
     */
    public static function handle_delivery_outcome( $http_args, $response, $duration, $arg, int $webhook_id ): void {
        $webhook = wc_get_webhook( $webhook_id );

        if ( ! $webhook instanceof WC_Webhook ) {
            return;
        }

        $circuit_key  = self::get_circuit_key( $webhook->get_delivery_url() );
        $is_wp_error  = is_wp_error( $response );
        $circuit_open = $is_wp_error && self::OPEN_ERROR_CODE === $response->get_error_code();
        $code         = $is_wp_error ? 0 : (int) wp_remote_retrieve_response_code( $response );

        // Success (2xx): reset the circuit.
        if ( $code >= 200 && $code < 300 ) {
            delete_transient( $circuit_key );
            return;
        }

        // Permanent client errors are not retried, except 408 and 429.
        if ( $code >= 400 && $code < 500 && 408 !== $code && 429 !== $code ) {
            return;
        }

        // Count real endpoint failures only. A short-circuited call never reached the
        // endpoint, and counting it would renew the 120-second window on every skipped
        // delivery, so the circuit would never close while traffic keeps arriving.
        if ( ! $circuit_open ) {
            set_transient( $circuit_key, (int) get_transient( $circuit_key ) + 1, self::CIRCUIT_COOLDOWN );
        }

        $attempt = isset( $http_args['headers']['X-WC-Webhook-Attempt'] )
            ? (int) $http_args['headers']['X-WC-Webhook-Attempt']
            : 1;

        if ( $attempt >= self::MAX_ATTEMPTS || ! function_exists( 'as_schedule_single_action' ) ) {
            return;
        }

        // Exponential backoff with jitter: (2 ^ attempt) * 30s + 0-15s.
        $delay_seconds = ( ( 2 ** $attempt ) * 30 ) + random_int( 0, 15 );

        as_schedule_single_action(
            time() + $delay_seconds,
            self::RETRY_HOOK,
            [ $webhook_id, $arg, $attempt + 1 ],
            self::RETRY_GROUP
        );
    }

    /**
     * Retry worker: re-deliver with the current attempt number in the header.
     *
     * @param int   $webhook_id Webhook ID.
     * @param mixed $arg        Payload identifier (for example, an order ID).
     * @param int   $attempt    Attempt number for this delivery.
     */
    public static function execute_backoff_retry( int $webhook_id, $arg, int $attempt ): void {
        $webhook = wc_get_webhook( $webhook_id );

        if ( ! $webhook instanceof WC_Webhook || 'active' !== $webhook->get_status() ) {
            return;
        }

        $inject_attempt_header = static function ( array $http_args ) use ( $attempt ): array {
            $http_args['headers']['X-WC-Webhook-Attempt'] = (string) $attempt;

            return $http_args;
        };

        // Priority 99 so this runs after the timeout filter has initialized the header.
        add_filter( 'woocommerce_webhook_http_args', $inject_attempt_header, 99, 1 );

        try {
            $webhook->deliver( $arg );
        } finally {
            remove_filter( 'woocommerce_webhook_http_args', $inject_attempt_header, 99 );
        }
    }
}

WebhookCircuitBreaker::init();

How the pieces interact

Failure counting still happens in core. Every failed delivery, including a short-circuited one, still passes through WC_Webhook::deliver() and increments failure_count. That is why the threshold filter is not optional: with the default of 5, the circuit breaker would open and WooCommerce would disable the webhook in the same minute. Core resets the counter to zero on a successful delivery, so after recovery the numbers return to normal on their own.

The breaker is a sliding window. The transient holds a per-host counter with a 120-second TTL that is renewed on each real failure. After five failures the breaker opens. Once the host has gone 120 seconds without a counted failure, the transient expires and the next delivery goes through as a probe. A success deletes the counter; another failure starts counting again. This implementation approximates the three traditional circuit breaker states: closed, open, and half-open. In a full half-open circuit breaker state, a limited number of probe requests are allowed after the cooldown period to determine whether the remote endpoint has recovered. The simplified transient-based implementation above permits requests again after expiration but does not enforce an exclusive recovery probe. If your site has no persistent object cache, these transients live in wp_options—which is acceptable at this write rate as long as you maintain regular wp_options transient hygiene, though a persistent Redis cache makes the counter cheaper.

Short-circuited calls skip the counter but still get retried. The upstream research version incremented the counter for every WP_Error, including the breaker's own wc_webhook_circuit_open error. Because each increment rewrites the transient with a fresh 120-second TTL, a store with steady order traffic would keep the circuit open indefinitely, even after the ERP recovered. The $circuit_open check above prevents that while still scheduling the delayed retry.

The backoff schedule. The delay for each attempt is (2 ** attempt) * 30 seconds plus 0–15 seconds of jitter:

Failed attempt Next attempt runs after Cumulative wait (approx.)
1 60–75 s ~1 min
2 120–135 s ~3 min
3 240–255 s ~7 min
4 480–495 s ~15 min
5 960–975 s ~31 min
6 none (final attempt) n/a

The window covers roughly a 30-minute outage. Jitter spreads the retries of events that failed in the same second, so a recovering ERP is not hit by every queued order at once. If your SLA needs a longer window, raise MAX_ATTEMPTS or cap the exponent and add a ceiling to the delay. Exponential backoff with jitter is a distributed systems resilience pattern that progressively increases the delay between retry attempts while introducing randomized scheduling intervals. The implementation above uses additive jitter of 0–15 seconds to reduce synchronized webhook retries, while a bounded attempt limit prevents indefinite retry loops and uncontrolled queue growth.

Retries run in their own group. Scheduling into wc-webhooks-retry lets you monitor, pause, or purge retry actions independently of woocommerce_deliver_webhook_async and of payment and subscription hooks. Positional arguments are used for the scheduled action, so the callback receives $webhook_id, $arg, and $attempt in order on any PHP version.

The retry worker rebuilds the payload. deliver( $arg ) generates the payload at retry time, so the receiver gets the order as it is now, not as it was at the first attempt. The try / finally block guarantees the temporary header filter is removed even if deliver() throws, so the attempt number cannot leak into unrelated deliveries running in the same PHP process.

Operational Notes

  • Implement an idempotent webhook consumer. A retry can arrive after the original request was successfully processed but its HTTP response timed out. Just as inbound gateways require mutex serialization and idempotency against Stripe and PayPal webhook race conditions, your receiving ERP, 3PL, or middleware system should use durable idempotency keys and duplicate-event detection to prevent repeated business operations, such as duplicate shipment creation or inventory adjustments. Where possible, use a stable business-operation identifier rather than the webhook delivery ID, which may change between attempts. Use X-WC-Webhook-Attempt for logging and retry diagnostics.
  • Test with a controlled failure. Point a staging webhook at an endpoint that returns 503, trigger a few orders, and confirm that retries appear in the wc-webhooks-retry group, that failure_count rises without the status changing to disabled, and that the circuit opens after five failures. Then return 200 and confirm the counter resets.
  • Alert on the retry group. Pending or failed actions in wc-webhooks-retry older than your SLA are the signal that matters now, since core will no longer report anything.
  • Run a reconciliation job for exhausted retries. After the sixth attempt the event is dropped by this plugin. For fulfillment-critical flows, pair it with a periodic job that compares recent WooCommerce orders against the receiver's records.
  • Re-enable what already broke. If Query 1 shows webhooks disabled by an earlier outage, set them back to active in the WooCommerce webhook settings once the endpoint is healthy, and replay any events missed in the interval.

Summary

WooCommerce does not retry failed webhooks, a longer timeout does not fix it, and a bare exception makes it worse. The reliable pattern is a short timeout, a host-level circuit breaker that fails fast during an outage, and retries with exponential delay and jitter in a separate Action Scheduler group, with the core failure threshold raised so a maintenance window never disables an order or fulfillment integration. Deployed as a must-use plugin, it contains an ERP outage to a delayed delivery instead of a queue stall or a silently dead webhook.

0 Comments