AI Search Visibility
AI Crawler Management
Most AI crawler decisions are made by default settings or by accident. Strategic visibility into which bots are reading your site, intentional rules for what they can do, and ongoing monitoring as the landscape changes.
Deliverables
AI bot log audit
Forensic review of which AI bots are crawling your site, what they are doing, and how often. Documented with specifics.
Access strategy document
A documented decision framework: which AI engines get full access, which get limited, which get blocked, and the reasoning behind each call.
robots.txt and access rules deployed
Server-level and file-level rules for managing AI bot access, validated in production with test requests.
Monitoring dashboards setup
Ongoing visibility into AI bot activity through log-based monitoring. You see what is happening without waiting on me.
Bot landscape monitoring
Ongoing tracking of new AI crawlers as they emerge. Recommendations when the landscape changes.
Documented handover
Written runbook for managing AI crawlers going forward. Your team can run with it after the engagement closes.
Process
Discovery call
Thirty minutes. We walk through your site, your hosting, your current robots.txt state, and any concerns about AI crawler activity. By the end I know what scope fits.
Log audit
One to two weeks. Forensic analysis of which AI bots have been crawling your site, what they have been requesting, and the patterns of access. Output is a documented audit report.
Strategy and decisions
I work through the access decisions with you: which bots get full access, which get limited, which get blocked. Every call is documented with reasoning so future team members understand the choices.
Implementation
Deploying the rules: robots.txt updates, server-level rules where relevant, llms.txt declarations where they help. Validated in production with test requests against each AI bot user agent.
Monitoring and handover
Setting up ongoing monitoring so you have visibility into AI bot activity. Documented runbook for your team to manage the rules going forward.
Packages
AI Bot Audit
From £1,500
Forensic look at what is crawling your site
1 to 2 weeks
- Forensic log analysis of current AI bot activity
- Identification of every AI engine currently crawling
- Strategic recommendations document with reasoning
- Risk and opportunity assessment
- Deployment guide for your team to act on findings
Full Implementation
From £3,800
Audit, strategy, deployed rules, monitoring
3 to 5 weeks
- Everything in the audit
- Documented access strategy per AI engine
- robots.txt and server-level rules deployed
- Monitoring dashboards setup
- Validation testing across all AI bot user agents
- Written runbook for your team
Ongoing Management
From £600/month
AI crawler activity, watched and adjusted
Monthly cadence
- Monthly bot activity review and reporting
- Rule adjustments as the AI landscape changes
- New AI crawler detection and recommendations
- Quarterly strategy review and refresh
- Direct email access for AI bot questions
Case Studies
Halewood Editorial
Differentiated AI bot access for an editorial archive
Editorial site had aggressive AI bot traffic from multiple engines, no visibility into which were driving citation back to the site and which were just training. Built a log analysis pipeline, identified the citation-driving engines, established differentiated access rules and monitoring.
Outcome:
Cendric
Strategic access rules for product documentation
B2B SaaS was concerned pricing pages and product documentation were being scraped by AI engines without driving citation. Implemented strategic access: full access for citation-driving engines (Perplexity, ChatGPT), restricted access for training-only crawlers, monitoring across both.
Outcome:
Aldernode
Server-level rate limiting for aggressive AI bots
E-commerce site had AI bot traffic affecting server response times during peak hours. Mapped the bot patterns through log analysis, implemented rate-limiting rules at the server level, configured robots.txt for specific AI bot user agents.
Outcome:
FAQs
Do you need access to my server logs?
Ideally yes. The audit work depends on access to your access logs (Apache, Nginx, Cloudflare, etc.). If raw log access is not possible, I can work with summary data from your CDN, hosting provider, or analytics platform - though the analysis is less granular. We confirm what is available during discovery.
Can you do this without server access (using only SaaS dashboards)?
Yes, with caveats. CDN logs (Cloudflare, Fastly), hosting dashboards (Vercel, Netlify), and analytics platforms (GA4 with bot reporting) can give you bot visibility - just less detail than raw server logs. The audit recommendations adapt to what data you have.
Will managing AI crawlers affect my SEO?
It can affect AI Search visibility, yes - that is the whole point. The work is about making intentional decisions about access. Done well, it improves citation. Done badly (such as blocking everything by default), it eliminates AI Search visibility entirely. The strategy step is where this gets right.
Which AI bots should I allow and which should I block?
Generally: allow the engines that drive citation back to your site (ChatGPT, Perplexity, Claude). Make a deliberate choice about training-only crawlers (CCBot, Google-Extended for training, and others). Restrict aggressive bots that crawl without giving anything back. The specifics depend on your business model and content strategy.
What is the difference between Googlebot and Google-Extended?
Googlebot is Google’s web crawler for traditional search. Google-Extended is the AI bot for training Gemini and other Google AI products. They are separate user agents. You can allow Googlebot for search while blocking Google-Extended for AI training, which many publishers now do.
Do you handle the actual server changes or do I need a sysadmin?
I draft and document the rules. Deployment depends on your stack. For robots.txt and llms.txt, your team can usually deploy directly. For server-level rules (.htaccess, Nginx config, Cloudflare rules), your sysadmin or DevOps team handles deployment. I provide deployment instructions and validate the result.
How often do you update the rules?
Audit and initial implementation: one-time work. Ongoing management engagements: monthly check-ins to review activity, plus quarterly rule reviews as the AI engine landscape changes. New AI crawlers emerge regularly. The rules need to keep up.
Do I keep the documentation and rules?
Yes. Everything I deliver is yours: the audit document, the strategy framework, the deployed rules, the monitoring setup, the runbook. Your team can manage the rules going forward.