Rattlesnakes By Mail

Daily snapshot 2026-11-19

2026-11-19 UTC

The numbers come from one website. They describe what these crawlers did here, not across the web.

Period: 2026-11-19T00:00:00Z to 2026-11-20T00:00:00Z (00:00Z to 24:00Z). Generated 2026-11-20T03:18:20.023Z. Schema version 1. Licence CC BY 4.0. Cite as: Benes, Peter. Rattlesnakes By Mail: A Field Guide to the Crawlers of the Web. benesthemenace.com. Method: https://rattlesnakesbymail.com/method

Requests by crawler on 2026-11-19Requests on 2026-11-19 by crawler, against each crawler's 28-day median. Googlebot: 85 requests, 28-day median 47; Bingbot: 47 requests, 28-day median 38; Applebot: 4 requests, 28-day median 0; GPTBot: 4 requests, 28-day median 4; OAI-SearchBot: 4 requests, 28-day median 6.GooglebotGooglebot: 85 requests28-day median 4785BingbotBingbot: 47 requests28-day median 3847ApplebotApplebot: 4 requests4GPTBotGPTBot: 4 requests28-day median 44OAI-SearchBotOAI-SearchBot: 4 requests28-day median 64
Bars show requests on 2026-11-19. The black tick on each row marks that crawler's 28-day median.

Facts

Records selected at random for 2026-11-20

Ten records from the field guide are chosen at random each day. The selection and how it was drawn are listed in /snapshot.json.

Crawlers

CrawlerRequestsVerified shareDistinct paths28-day medianSweep dayFirst pathrobots.txtsitemap
Googlebot85100%2647no/robots.txt221
Bingbot47100%3938no/10
Applebot4100%20no/robots.txt20
GPTBot4100%44no/sitemap-index.xml03
OAI-SearchBot4100%16no/robots.txt40

MCP calls

ClientToolCalls
BrickBlueBotget_snapshot1
BrickBlueBotidentify_user_agent1
BrickBlueBotlist_changes1
BrickBlueBotlookup_crawler1
RadialScout__radial_auth_probe__1
mcp-index__mcp_index_nonexistent__1
mcpyp-incrementalidentify_user_agent1
mcpyp-incrementallist_changes1
mcpyp-incrementallookup_crawler1
rokmcp-probelist_changes1
verifymcp-probe__verifymcp_auth_probe_b34241ddfcf805bc__1

Snapshot pickup

For each of the previous seven snapshots, the first fetch by each verified crawler, in minutes after publication, and the c tag on that fetch.

SnapshotCrawlerMinutes after publicationTag
2026-11-18Googlebot1,193none
2026-11-18GoogleOther1,193p
2026-11-18Bingbot1,325none
2026-11-17Bingbot293none
2026-11-17Googlebot809none
2026-11-17GoogleOther886h
2026-11-16OAI-SearchBot78none
2026-11-16GPTBot78h
2026-11-16Bingbot293none
2026-11-16Googlebot1,894none
2026-11-16GoogleOther2,244p
2026-11-15OAI-SearchBot77none
2026-11-15GPTBot77h
2026-11-15Bingbot292none
2026-11-15Googlebot866none
2026-11-15GoogleOther1,479p
2026-11-14OAI-SearchBot77none
2026-11-14GPTBot77h
2026-11-14Bingbot293none
2026-11-14Googlebot1,135none
2026-11-14GoogleOther2,995none
2026-11-13OAI-SearchBot78none
2026-11-13GPTBot78h
2026-11-13Bingbot293none
2026-11-13Googlebot628none
2026-11-13GoogleOther881none
2026-11-12Bingbot292none
2026-11-12Googlebot840none
2026-11-12OAI-SearchBot1,517none
2026-11-12GPTBot1,517p

Open questions

If you can answer one of these with a source, send a note: POST https://rattlesnakesbymail.com/notes with target question:<id>, where <id> is the number shown with the question, and body=<your answer and its source>.

Ask for something this snapshot lacks

What data about web crawlers did you look for and not find? Search https://rattlesnakesbymail.com/search?q=your+question. Searches with no match are recorded as gaps. A gap is published at /questions only when an editor publishes it; nothing is published automatically. To suggest something this daily snapshot should include, send a note with target snapshot:261119.

To send it, POST https://rattlesnakesbymail.com/notes as a web form or as JSON with target=snapshot:261119 and body=<your note>. The same note can be sent with target_type=snapshot and target_id=261119. The optional author_claim field says who is writing: a person, a model or an agent.

Links

Next snapshot: 2026-11-20