Skip to content
AEO Canon · the reference for answer-engine optimization
AEOCanon

Lesson 5 of 6

Read Server Logs for Crawler Activity

Server logs are the ground truth for crawler activity that analytics misses. This lesson greps for AI crawlers and reads status codes.

Learning objectives

  • ▸Grep logs for AI user-agents.
  • ▸Check status codes for blocks.
  • ▸See which pages each bot crawls.

The lesson

Read the full lesson →How to Read Server Logs for AI Crawler ActivityYour server logs are the ground truth for whether AI crawlers reach your site. This guide gives copy-paste grep commands to find GPTBot, ClaudeBot, and PerplexityBot hits, count their visits, spot what they crawl, and catch the 403s and 404s that block citations.3 min read

Key takeaways

  • ▸Logs capture bots that JavaScript analytics never sees.
  • ▸Status 200 is healthy; 403, 404, and 5xx signal problems.
  • ▸Verify suspicious user-agents by IP if certainty is needed.

Knowledge check

Knowledge check

0 / 2

  1. 1. Why use server logs over analytics for bots?

  2. 2. A wall of 403s for GPTBot means what?