Technical SEO Engineering
Log File Analysis
Search Console shows you a sample. Analytics shows you users, not bots. Logs show you the truth: which crawlers are actually fetching your site, what they are requesting, what they are getting back, and where your crawl budget is leaking.
Deliverables
Crawler activity audit
Full breakdown of which bots crawled your site, how often, where they spent time, and which sections they ignored. The forensic baseline.
Crawl budget analysis
Where your crawl budget actually goes versus where it should. Per-section breakdown showing high-value pages versus crawl waste.
Crawl waste identification
Duplicate URLs, parameter-heavy paths, filter combinations, and pagination depths that are eating crawl budget without delivering value.
AI bot activity report
Which AI engines accessed your site, what they fetched, and how often. The visibility layer most log analysis services skip entirely.
Recommendations document
Prioritised actions based on the actual findings: robots.txt updates, canonical fixes, internal link adjustments, server-level rules where needed.
Monitoring setup
Ongoing visibility into crawler patterns through dashboards or scheduled reports. You see what is happening without needing me on every request.
Process
Discovery call
Thirty minutes. We walk through your site, your hosting setup, what log access is possible, and the questions you want answered. By the end I know whether the data we have is enough for the questions you have.
Data ingestion and processing
One week. Receiving logs (typically 30 to 90 days of access logs), parsing the data into a structured format, normalising user agents and bot identification. I work with raw logs, CDN logs, or hosting platform exports.
Forensic analysis
One to two weeks. Identifying crawler patterns: which bots, what they crawled, response codes, response times, crawl depth, parameter handling, AI bot share. Each finding documented with the data behind it.
Recommendations and reporting
Findings documented with prioritised actions. Crawl waste fixes, robots.txt updates, canonical changes, server-level rules. Every recommendation tied back to specific evidence in the logs.
Monitoring setup
For ongoing engagements: dashboards or scheduled reports for continuous crawler visibility. Your team sees the data without needing me on every request.
Packages
Log File Audit
From £1,800
One-time forensic look at your crawler activity
2 to 3 weeks
- Full crawler activity audit across 30 to 90 days of logs
- Per-bot breakdown including Googlebot, Bingbot, and AI bots
- Crawl budget waste identification
- Response code distribution and error patterns
- Documented findings with prioritised recommendations
Audit & Implementation Guidance
From £4,500
Findings plus the plan to act on them
4 to 6 weeks
- Everything in the audit
- Detailed implementation plan with engineering effort estimates
- robots.txt updates drafted and validated
- Canonical strategy recommendations with specifics
- Implementation guidance and review across the rollout
- Post-implementation validation with follow-up log analysis
Ongoing Monitoring
From £700/month
Continuous crawler visibility
Monthly cadence
- Monthly log analysis and crawler activity report
- Trend tracking across Googlebot, Bingbot, and AI bots
- Anomaly detection for unusual crawler patterns
- Quarterly deep-dive reviews with strategic recommendations
- Direct email access for log analysis questions
Case Studies
Halewood Editorial
Editorial archive crawl efficiency forensics
Editorial site with a deep article archive was seeing slow indexing of new content. Log analysis revealed Googlebot was spending most of its budget re-crawling old archive pages while ignoring the new ones. Built a robots and canonical strategy that redirected crawl attention to fresh content.
Outcome:
Cendric
Pre-migration crawler baseline
B2B SaaS was three months from a major replatform. Captured the full pre-migration crawler baseline across Googlebot, Bingbot, and AI bots. Established the dataset against which post-migration recovery would be measured. Critical for proving the migration did not damage crawl efficiency.
Outcome:
Aldernode
Faceted navigation crawl waste audit
E-commerce site had crawl budget being consumed by filter URL combinations producing near-duplicate pages. Log analysis split the activity per bot and per parameter, identified the worst-offending combinations, designed targeted robots and canonical rules that recovered budget for important product pages.
Outcome:
FAQs
Do you need access to my raw server logs?
Ideally yes. Raw access logs from Apache, Nginx, or your web server give the most granular data. If raw log access is not possible, I work with CDN logs (Cloudflare, Fastly), hosting platform exports (Vercel, Netlify), or analytics platforms with bot reporting. The analysis adapts to the data available.
What if we do not have raw log access?
Many sites do not. CDN-level logs from Cloudflare or similar usually capture enough crawler activity for meaningful analysis. Hosting platforms like Vercel and Netlify expose useful subsets. Even GA4 with bot filtering disabled can supplement the picture. We confirm what is realistically available during discovery.
What is the difference between log analysis and Search Console?
Search Console shows you what Google chooses to surface, sampled. Log analysis shows you what every crawler actually did, complete. Search Console tells you what Google saw and how it interpreted things. Logs tell you what every bot requested and what your server returned. Both are useful. Logs are forensic.
How much log data do you need?
Minimum: thirty days of access logs. Ideal: ninety days. Less than thirty days makes patterns harder to identify. Longer windows let us spot weekly, monthly, and seasonal patterns. For pre-migration baselines, we capture as much as you can provide.
What format do logs need to be in?
Common formats work directly: Apache Common Log, Apache Combined, Nginx default, Cloudflare logs, Fastly logs. Custom formats need a short discovery step to map fields. The work I do parses the data into a standard structure either way, so format flexibility is built in.
Do you analyse only Googlebot or every crawler?
Every crawler. Googlebot, Bingbot, the AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot), and any other bots fetching your content. The AI bot layer is what most log analysis services skip. For modern crawl strategy, you need the full picture.
How long does the analysis take?
One-time audit: two to three weeks from receiving logs to delivering findings. Larger sites with hundreds of millions of log entries may take longer. We scope realistically during discovery.
Do I keep the analysis and reports?
Yes. Everything I deliver is yours: the audit document, the parsed dataset (where useful), the recommendations document, any monitoring setup. Your team can rerun similar analyses going forward using the same framework.