StackFox/Documentation
Dashboard

Installation Guide

Track AI bot traffic to your website. Server-side integration is recommended because most AI crawlers don't execute JavaScript.

Quick Start

  1. 1. Create an account and get your API key
  2. 2. Choose your framework below and copy the code
  3. 3. Replace sf_YOUR_API_KEY with your actual key
  4. 4. Deploy and watch the data flow in your dashboard

Select Your Framework

1. Setup

# Install dependencies (optional - for type hints)
npm install @stackfox/sdk

# Or just copy the SDK to your project
curl -o lib/stackfox.ts https://stackfox.co/sdk/stackfox.ts

2. Integration Code

// middleware.ts (at your project root)
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';

const STACKFOX_KEY = 'sf_YOUR_API_KEY';
const STACKFOX_ENDPOINT = 'https://stackfox.co/api/v1/events';

// AI bot patterns
const BOT_PATTERNS = [
  /gptbot/i, /chatgpt-user/i, /oai-searchbot/i,
  /claudebot/i, /claude-web/i, /anthropic/i,
  /google-extended/i, /googleother/i,
  /perplexitybot/i, /meta-externalagent/i,
  /bytespider/i, /ccbot/i, /cohere-ai/i,
  /amazonbot/i, /applebot/i, /bingbot/i,
];

function isAIBot(ua: string): boolean {
  return BOT_PATTERNS.some(p => p.test(ua));
}

export function middleware(request: NextRequest) {
  const userAgent = request.headers.get('user-agent') || '';

  // Only track AI bot traffic
  if (isAIBot(userAgent)) {
    // Fire and forget - don't block the response
    fetch(STACKFOX_ENDPOINT, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-SF-Key': STACKFOX_KEY,
      },
      body: JSON.stringify({
        events: [{
          userAgent,
          path: request.nextUrl.pathname,
          method: request.method,
          timestamp: new Date().toISOString(),
        }]
      }),
    }).catch(() => {});
  }

  return NextResponse.next();
}

// Match all routes except static files
export const config = {
  matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'],
};

Notes

  • Place middleware.ts at your project root (same level as app/ or pages/)
  • Works with both App Router and Pages Router
  • Bot requests are tracked without blocking the response
  • Replace sf_YOUR_API_KEY with your actual API key from the dashboard

API Reference

POST /api/v1/events

Ingest bot visit events

Headers

X-SF-KeyYour API key (required)
Content-Typeapplication/json

Request Body

{
  "events": [
    {
      "userAgent": "GPTBot/1.0 (+https://openai.com/gptbot)",
      "path": "/blog/article",
      "method": "GET",
      "timestamp": "2024-01-15T10:30:00Z"  // optional
    }
  ]
}

Response

{
  "success": true,
  "processed": 1
}

Limits

  • • Maximum 100 events per request
  • • User-Agent limited to 500 characters
  • • Path limited to 2000 characters

Detected AI Bots

StackFox automatically identifies these bots and categorizes them by purpose:

Training

  • GPTBot (OpenAI)
  • ClaudeBot (Anthropic)
  • Google-Extended
  • CCBot (Common Crawl)
  • Meta-ExternalAgent
  • Bytespider (ByteDance)
  • Amazonbot
  • Cohere-AI

Search

  • OAI-SearchBot
  • PerplexityBot
  • YouBot (You.com)
  • Phind
  • Bingbot
  • Applebot

Assistant

  • ChatGPT-User
  • Claude-Web

Agent

  • Operator (OpenAI)
  • Computer-Use (Anthropic)

FAQ

Why server-side instead of client-side JavaScript?

AI crawlers like GPTBot and ClaudeBot make HTTP requests and parse the HTML response - they don't execute JavaScript. Server-side middleware intercepts these requests directly, ensuring you capture all bot traffic.

Does this slow down my site?

No. The tracking request is fire-and-forget - it doesn't block your response. We use async fetch calls that complete in the background, adding zero latency to your users.

How do I verify it's working?

After deploying, send a test request with a bot user-agent:

curl -H "User-Agent: GPTBot/1.0" https://yoursite.com/any-page

Check your StackFox dashboard - the visit should appear within seconds.

What data do you collect?

Only what's needed for bot tracking: User-Agent, path, HTTP method, and timestamp. We don't collect IP addresses, cookies, or any personally identifiable information.

Ready to track AI bot traffic?

Create an account to get your API key and start monitoring in minutes.

StackFox - Monitor & Control AI Bot Traffic