Understanding and Resolving 401 Unauthorized and 403 Forbidden Errors

When the RapidLoad crawler cannot access or read your site, it may result in 401 (Unauthorized) or 403 (Forbidden) errors. These errors can occur due to several reasons. Below are the common causes and their resolutions:

1. Firewall Restrictions

Firewalls on your site may block the RapidLoad user agent. To resolve this, whitelist the RapidLoad user agent:

RapidLoad/1.0

2. CloudFlare Bot Mode

If CloudFlare is installed on your site with the "Bot Mode" feature enabled, it may block the RapidLoad crawler. This is because RapidLoad sends multiple requests to optimize the files, and CloudFlare may identify these requests as bot activity. To resolve this issue:

  • Disable the CloudFlare Bot Mode.
Note: The RapidLoad team has contacted CloudFlare multiple times to request the whitelisting of RapidLoad, but we have not received a response. Users may also reach out to CloudFlare to request whitelisting.

3. Authentication required to preview the website

Authentication Filters for Resolving 401 and 403 Errors in RapidLoad

To ensure that the RapidLoad crawler can properly access and optimize your site, we have implemented two filters that you can add to your functions.php file. These filters address common authentication issues that may result in 401 and 403 errors.

1. Header Authentication

To add a header for authentication, use the following filter:

add_filter('uucss/cache/bust', function($args) {
    $args[] = [
        'type' => 'header',
        'rule' => 'Authorization:username@password'
    ];
    return $args;
});

In this filter:

  • A header is added to the request.
  • The key is Authorization.
  • The value is username@password.

2. Basic Access Authentication

add_filter('uucss/cache/bust', function($args) {
    $args[] = [
        'type' => 'urlAuth',
        'rule' => 'username:password'
    ];
    return $args;
});

In this filter:

  • A header is added to the request.
  • The key is Authorization.
  • The value is Basic + Buffer.from('username:password').toString('base64').

By implementing these filters in your functions.php file, you can effectively resolve 401 and 403 errors, ensuring that the RapidLoad crawler can access and optimize your site. If you encounter any further issues or require additional assistance, please feel free to reach out to our support team.

Table of Contents