Bypassing Bot Protection

Many websites use bot protection services to block automated traffic. When testing these sites, you'll encounter CAPTCHAs, challenge pages, or outright blocks. Test-Lab can bypass these protections using custom headers or cookies - the same approach used by internal QA teams.

How It Works

Most bot protection services support bypass mechanisms for legitimate automated testing:

1. Allowlist headers - Custom headers that signal "this is an authorized test"
2. Bypass tokens - Pre-generated tokens that skip challenge pages
3. Service tokens - API credentials that authenticate the request source

Test-Lab injects these headers/cookies into every browser request before the AI agent starts testing. The protection service sees the bypass signal and lets the request through.

CDN/Edge Platforms

Cloudflare

Cloudflare offers several protection layers: Turnstile (CAPTCHA), Bot Management, and WAF rules.

Bypass options:

Example - Cloudflare Access service token:

{
  "name": "CF-Access-Client-Id",
  "value": "your-service-token-id.access"
}

{
  "name": "CF-Access-Client-Secret",
  "value": "your-service-token-secret"
}

To create a service token: Cloudflare Dashboard → Zero Trust → Access → Service Auth → Create Service Token.

Fastly (Signal Sciences)

Fastly's Signal Sciences provides WAF and bot detection.

Bypass options:

Example:

{
  "name": "X-SigSci-Bypass",
  "value": "your-bypass-token"
}

Contact your security team for the specific header name and token value.

Akamai Bot Manager

Akamai's Bot Manager uses behavioral analysis and challenge pages.

Bypass options:

Example:

{
  "name": "Akamai-BM-Verification",
  "value": "your-verification-token"
}

AWS CloudFront + WAF

AWS WAF rules can block bot traffic at the CloudFront edge.

Bypass options:

Example - Custom bypass header:

{
  "name": "X-Test-Bypass",
  "value": "your-secret-token"
}

Create a WAF rule that allows requests containing this header to bypass other rules.

Sucuri

Sucuri provides WAF and DDoS protection.

Bypass options:

CAPTCHA Services

Google reCAPTCHA

reCAPTCHA v2 and v3 are embedded in forms to verify human users.

reCAPTCHA v2/v3 (free tier):

The free tier has no IP allowlist or production bypass mechanism. The only option is test keys for development:

• Site key: 6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI
• Secret key: 6LeIxAcTAAAAAGG-vFI1TnRWxMZNFuojJ4WifJWe

reCAPTCHA Enterprise:

Enterprise tier supports proper bypass for automated testing:

hCaptcha

hCaptcha provides CAPTCHA challenges similar to reCAPTCHA.

Bypass options:

Example - Accessibility cookie:

{
  "name": "hc_accessibility",
  "value": "your-accessibility-token",
  "domain": ".yourdomain.com"
}

Request an accessibility token from hCaptcha support or your hCaptcha dashboard.

Dedicated Bot Protection Services

DataDome

DataDome provides real-time bot detection across web, mobile, and API.

Bypass options:

Example:

{
  "name": "X-DataDome-Bypass",
  "value": "your-bypass-token"
}

PerimeterX (HUMAN)

PerimeterX (now HUMAN Security) uses behavioral analysis and challenge pages.

Bypass options:

Example - Bypass cookie:

{
  "name": "_pxbypass",
  "value": "your-bypass-token",
  "domain": ".yourdomain.com"
}

Kasada

Kasada provides bot mitigation with proof-of-work challenges.

Bypass options:

Configuring in Test-Lab

Once you have bypass headers or cookies from your security team, configure them in Test-Lab:

Project-Level (applies to all test plans)

1. Go to Projects in the admin dashboard
2. Click Edit on your project
3. Expand Advanced Settings
4. Add headers under HTTP Headers or cookies under Cookies
5. Click Save

Test Plan-Level (overrides project settings)

1. Go to Test Plans in the admin dashboard
2. Click Edit on your test plan
3. Expand Advanced Settings
4. Add headers or cookies (these override project-level values)
5. Click Save

Runtime via API (for dynamic tokens)

Pass headers or cookies when triggering tests:

curl -X POST https://test-lab.ai/api/v1/run \
  -H "Authorization: Bearer tl_xxxxx" \
  -H "Content-Type: application/json" \
  -d '{
    "testPlanId": 123,
    "headers": [
      { "name": "CF-Access-Client-Id", "value": "token-id.access" },
      { "name": "CF-Access-Client-Secret", "value": "token-secret" }
    ]
  }'

See Cookie & Header Authentication for complete configuration details.

CI/CD Integration

For dynamic bypass tokens that rotate or expire, generate them in your CI pipeline:

- name: Get bypass token
  id: bypass
  run: |
    TOKEN=$(curl -X POST https://your-security-api/generate-bypass-token \
      -H "Authorization: Bearer ${{ secrets.SECURITY_API_KEY }}" \
      | jq -r '.token')
    echo "token=$TOKEN" >> $GITHUB_OUTPUT

- name: Run Test-Lab tests
  run: |
    curl -X POST https://test-lab.ai/api/v1/run \
      -H "Authorization: Bearer ${{ secrets.TESTLAB_API_KEY }}" \
      -H "Content-Type: application/json" \
      -d '{
        "testPlanId": 123,
        "headers": [
          { "name": "X-Bypass-Token", "value": "${{ steps.bypass.outputs.token }}" }
        ]
      }'

Troubleshooting

Still seeing challenge pages?

1. Verify header names - Header names are case-sensitive for some services
2. Check token validity - Tokens may have expired or be environment-specific
3. Confirm configuration - Ensure headers are set at the right level (project vs test plan)
4. Test manually - Use browser DevTools to verify headers are sent correctly

Protection service not recognizing bypass?

• Some services require both headers AND cookies
• IP allowlisting may be required alongside headers
• Check if your staging environment has different protection rules than production

Next Steps