A missing citation is not yet an access fault
If an assistant cannot open a specific page, record that exact retrieval problem and ask your website provider to check access. If an assistant simply does not mention your business, first determine whether it searched the web and what it actually cited. That result alone does not prove your site is blocked.
This worksheet is for a particular URL that may be inaccessible to a search crawler or user-triggered retrieval. The AI-search guide owns the broader business-fact and answer-observation work. Fixing access makes content available for consideration; it does not select your business for an answer.
Write down the page URL, question, platform, date and reported error. Keep a failed retrieval separate from an answer that used other sources successfully.
Identify the bot whose job you are testing
| Agent | Published purpose | What to ask the provider to inspect |
|---|---|---|
| OAI-SearchBot | ChatGPT search discovery | Search crawl rules and published OpenAI search IP access |
| ChatGPT-User | User-triggered page access | Actual retrieval response; this is not the automatic search crawl control |
| Claude-SearchBot | Claude search indexing/relevance | Its robots rule and hosting response |
| Claude-User | Claude user-triggered retrieval | Its robots rule and the response to a permitted retrieval |
| GPTBot / ClaudeBot | Potential training collection | The owner's separate training preference |
OpenAI distinguishes OAI-SearchBot from GPTBot and says robots.txt may not apply to ChatGPT-User actions. Anthropic says its bots honour robots.txt and do not bypass CAPTCHAs. Do not change a training preference just because you want search discovery. For your developer, use the live vendor documentation rather than a copied list of old IP addresses.
Sources: OpenAI: crawler purposes and access, Anthropic: crawler purposes and controls
Copy this fault report to your website provider
Fill this from the affected page and the actual observation. Keep credentials and private customer URLs out of the report.
- Public URL: exact address ___; final hostname after any redirect ___.
- Expected content: actual service/contact details the page should show ___.
- Human check: signed-out browser result ___; time ___; login/challenge/redirect shown ___.
- Assistant observation: platform ___; question or URL request ___; search/retrieval attempted ___; exact result ___; time ___.
- Requested bot purpose: search or user retrieval ___; relevant agent ___.
- Provider evidence requested: current robots rules for this host; permitted bot's actual response; any CDN/firewall challenge; redirect chain; textual content returned.
- Owner preference: public search access ___; user retrieval ___; separate training preference ___.
- Proposed correction: exact rule/page change ___; who approves it ___; how to restore it ___.
- Readback: same public URL and same affected access path checked ___; actual result ___.
A developer sending a request with a bot-like user-agent string is a preliminary test, not proof of a genuine vendor request. Where logs identify a vendor crawler, verify its origin using the current official IP information before making a targeted hosting change. Do not remove unrelated security controls merely to make a broad test pass.
Sources: OpenAI: crawler purposes and access, Anthropic: crawler purposes and controls
Fictional fault: public page replaced by a challenge
Fictional troubleshooting example. A roofer's service page opens in the owner's browser, but a particular user retrieval reports that it cannot access the page. The owner saves the URL, time and result. The website provider checks the affected retrieval path and discovers it receives a challenge page instead of the service text.
The provider identifies the relevant request and reviews the rule responsible. With the owner's approval, the provider corrects that specific access problem and checks the same path again. The receipt records “service text returned” rather than “Claude recommends this roofer”. If no affected request can be located, the status remains unresolved; an unrelated browser success is not substituted for it.
If the retrieved page contains an old phone number, the next job is to correct the public business fact. If the page retrieves successfully but an answer chooses other sources, return to the broader AI-search observation sheet instead of weakening more hosting rules.
Handle Google AI visibility as a separate check
For Google's AI Overviews and AI Mode, Google says a supporting page must be indexed and eligible for a snippet. It recommends ordinary SEO fundamentals, including crawl access, internal links and important content in text. It requires no special AI file or schema.
Ask your website provider to inspect the relevant page in Search Console if Google access or indexing is the issue. Keep that evidence separate from ChatGPT and Claude retrieval results. A fix for one agent is not evidence that every platform has crawled the page.
After access is working, use a real-job brief to give visitors useful, evidenced information. Keep the actual customer calls and resulting work in Tradehand's business app. An access test, a citation, a website visit and a booking each need their own observation.
