Skip to content

How do I see which AI crawlers visit my site?

Connect Cloudflare with a read-only token, or upload a server log, to see which AI crawlers and search bots read which pages.

What you get

Analytics > AI crawler logs shows which AI crawlers and search bots asked for pages on your site: which bot, which pages, on which days, and what your site answered (for example a page that was not found). It also checks each visit against the address ranges the bot’s owner publishes, where the owner publishes any, so a visitor only pretending to be a bot is left out of the totals.

A visit means a page was fetched. It does not mean an assistant showed or cited the page.

No visits from a crawler you expected? Check that your site lets it in with the free AI crawler access checker: it reads your robots.txt, the page’s noindex and your llms.txt.

Nothing is shown until your own data arrives. PromptMentions does not estimate bot visits.

Connect Cloudflare

If your site runs through Cloudflare, this is the quickest way: no log files and nothing to install. You create a read-only token in Cloudflare, and PromptMentions reads the bot visits Cloudflare already records, once an hour.

  1. In Cloudflare, open My Profile, then API Tokens, choose Create Token and then Create Custom Token.
  2. Under Permissions, add exactly one line: Zone, Analytics, Read. Do not add anything else, and nothing with Edit.
  3. Under Zone Resources, choose Include, Specific zone, and pick your site.
  4. Create the token and copy it. Then copy the Zone ID from your site’s Overview page in Cloudflare.
  5. In PromptMentions, open Analytics > AI crawler logs, go to Data sources, paste the Zone ID and the token, tick the box that says you are allowed to connect this zone, and choose Check token and connect.

Before saving, PromptMentions checks that the token can read your site’s analytics and that it cannot read your DNS records or zone settings or reach your other sites. A token that can do more is refused, so you can be sure a narrow one was stored. Cloudflare does not let anyone list everything a token is allowed to do, so please create it with the one permission above. The token is stored encrypted, is never shown again, and is only used to read visit counts.

Up to 7 days of earlier visits are read during the first hour or so, starting after anything already on the page. After that, each new hour is added about half an hour after it ends. An owner or editor with a confirmed email address can connect.

Which Cloudflare plans work

Cloudflare’s documentation says the data this uses is available on every Cloudflare plan, including Free. We have not tested every plan ourselves, so when you connect, PromptMentions asks Cloudflare what your site’s plan provides and tells you straight away if it is not enough. Your site has to be proxied through Cloudflare (the orange cloud) for Cloudflare to record its visits.

  • Cloudflare says it keeps about 31 days of this data, so connect before older visits age out.
  • Some plans leave out details, such as the visitor’s address. Without addresses, visits cannot be checked against the bot owners’ published ranges and count as unverified. After you connect, the page says exactly what your plan provides.
  • On busy sites Cloudflare counts a sample of requests and scales it up. When that happens the page says the counts are estimates.

Disconnect and delete the token

Under Disconnect Cloudflare, tick the box and choose Disconnect. The stored token is deleted straight away and the hourly reads stop. Visits already collected stay on the page. To finish, delete the token in Cloudflare too, under My Profile, API Tokens.

Upload a log file instead

If your site is not on Cloudflare, upload a log file under Upload a log file (up to 5 MB each, gzip is fine). These formats are read:

  • Apache or nginx access logs in the standard combined format.
  • Vercel: a log drain export, saved as one record per line or as one list.
  • Netlify: a log drain export of traffic logs.
  • Fastly: logs in the JSON format from Fastly’s logging guide, or a combined format that includes the user agent. Fastly’s default format leaves the user agent out, and without it bots cannot be told apart from people.
  • Amazon CloudFront: standard log files.
  • Cloudflare Logpush files, and plain JSON with one request per line.

Only lines from known AI crawlers and search bots are kept. Every other visitor’s line is counted and thrown away, and the part of an address after the question mark is never stored. Uploading the same file twice adds nothing twice.

Vercel, Netlify, Fastly and CloudFront are file formats you upload. There is no live connection to them: Cloudflare is the only one that connects.

Using more than one source

Once Cloudflare is connected it is the only source counted from that moment on. If you also upload a log that covers the same days, those lines are left out, so no visit is counted twice. Logs from before you connected are still added.

Questions

Can PromptMentions change anything in my Cloudflare account?

No. It only reads visit counts. It asks for a read-only token, refuses one that can read your DNS records or settings, and never sends Cloudflare anything except that one request for counts.

Who can see the bot visits?

People in your workspace who can open that brand or client. Someone limited to other clients cannot see it, and viewers cannot connect or disconnect.

Still need a hand?

Use the chat button in the corner of the page, send us a message or write to support@promptmentions.com. A person reads every message.

Looking for something else? See what the product does, pricing, how each figure is measured and how your data is protected.

Last checked against the product on 18 November 2026. All help articles