Yes, your robots.txt file now has an AI audience, although it is not one audience and it does not make one decision. A robots.txt file is a public, site-root text file that communicates crawler access preferences under the Robots Exclusion Protocol (REP); use it to decide separately whether a named bot may fetch pages for search answers, search indexing, or model training, then remember that it is neither an access-control system nor a promise that content cannot be found elsewhere. Request for Comments (RFC) 9309, the Robots Exclusion Protocol specification defines the protocol’s scope and limits.

For publishers asking how to unblock AI crawlers in robots.txt, the practical answer is narrow: add an allow rule for the specific crawler whose documented purpose you want, remove a more general rule that governs it, and verify that the pages remain reachable without authentication or another access barrier. OpenAI (OAI) calls its search crawler OAI-SearchBot and says publishers can allow it for appearance in ChatGPT search; it documents separate controls for GPTBot, its crawler associated with model training. OpenAI’s crawler documentation is the authority for those bot names and stated purposes.

Start with the permission you actually mean to grant

AI has made the old file feel newly consequential because a publisher may want discovery in one product but not another. Search discovery means a service may retrieve a page to produce or support a search result. Indexing means a search engine may retain information about a page for retrieval in results. Model training means content may be collected for developing or improving a model. Those are different purposes, and a rule that says “allow AI” is too imprecise to express a considered editorial policy. OpenAI distinguishes OAI-SearchBot from GPTBot, and Google Search Central’s crawler overview likewise distinguishes Google crawlers by function.

What common crawler permissions are intended to control
Crawler Documented role Publisher decision
OAI-SearchBot OpenAI crawler for ChatGPT search. Allow if ChatGPT search discovery is desired.
GPTBot OpenAI crawler associated with model training. Decide separately from search visibility.
Googlebot Google’s primary crawler for Search. Allow if Google Search crawling is desired.
The names and roles come from OpenAI’s crawler documentation and Google Search Central’s crawler overview; each rule is a purpose-specific policy choice, not a general AI setting.

A site may be visible in Google Search while blocking a particular OpenAI crawler, or it may permit a search-oriented crawler while declining training-oriented collection. Neither choice determines another company’s crawler behavior, because robots.txt rules are matched to the requesting user agent and each crawler has its own documented behavior. RFC 9309’s user-agent group model and OpenAI’s bot-specific guidance describe the two halves of that relationship.

Read your existing rules before adding a new one

A common obstacle is an existing catch-all group. In robots.txt, User-agent: * is a wildcard group for robots not matched by a more specific group; a named group lets you write a distinct policy for a particular crawler. Google explains that it selects the most specific applicable user-agent group, rather than combining a named group with the wildcard group. Google Search Central’s robots.txt introduction documents this matching behavior and the supported directives.

A broad block can therefore coexist with a deliberate exception. A publisher that wants ChatGPT search access while retaining its general block can express the choice plainly:

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /

The example is a policy pattern, not a magic incantation. It relies on the specific matching behavior documented by Google Search Central and on OpenAI’s published name for its search crawler in its crawler documentation. A site with generated files, multiple hosts, language directories, or a content delivery network should inspect the exact file served from every relevant origin before treating this snippet as complete.

Path rules deserve the same precision. Disallow: /private/ expresses a preference against crawling that path, whereas Allow: / covers the site root and its paths for that user-agent group. RFC 9309 defines how records are grouped and paths are matched, including its longest-match rule where allow and disallow records overlap. RFC 9309’s matching rules are worth reading before adding exceptions that overlap.

Do not confuse crawl permission with indexing or security

Robots.txt governs crawling preferences, not confidential access. The protocol warns that robots.txt is not a substitute for security measures, and a disallowed URL can still be known through links, logs, or other sources even if a cooperative crawler does not fetch it. Sensitive material belongs behind authentication and authorization, with robots.txt used only as an additional crawl instruction. RFC 9309’s security guidance supports that separation.

Indexing is a separate control surface. Google documents noindex as a robots meta tag or Hypertext Transfer Protocol response-header instruction for preventing a page from appearing in Google Search results, while noting that Googlebot must be allowed to crawl the page in order to see the rule. A blanket robots.txt block is therefore a poor substitute for an indexing decision when the goal is removal from Google results. Google Search Central’s noindex guidance explains the dependency.

A useful operating rule: use robots.txt to state which cooperative crawlers may fetch which paths; use page-level indexing directives where a search engine documents them; use authentication for information that must not be public. Those controls solve adjacent problems, not interchangeable ones. RFC 9309 and Google’s indexing guidance draw the distinction.

Make an AI-crawler decision that someone can maintain

The durable work is not the two-line edit. It is writing down which public content the organization wants discoverable through which services, who owns that decision, and what should trigger a review. Editorial, legal, engineering, and search teams may reach different answers, especially when an organization licenses content, publishes user material, or operates several domains. That is an editorial judgment, not a technical default.

Once the policy exists, make the implementation inspectable. Request the live /robots.txt response from the relevant production host, look for wildcard blocks and named crawler groups, add the narrowest rule that expresses the decision, then confirm that the file is publicly reachable and syntactically legible. Google documents that robots.txt belongs at the site root and is retrieved from the relevant host. Google’s robots.txt documentation covers those location and retrieval expectations.

One caveat belongs in the policy document: we do not know whether allowing a search crawler will produce visibility, citation, referral traffic, or a particular presentation in an AI interface. A robots.txt allow rule communicates crawl permission; it does not create a ranking guarantee or a commercial agreement. OpenAI frames its controls as publisher choices about its crawlers, while Google treats robots.txt as a crawling protocol rather than a ranking mechanism. OpenAI’s crawler documentation and Google Search Central’s robots.txt documentation support that restrained interpretation.

Key takeaways

Questions readers ask

How do I unblock AI crawlers in robots.txt?

Add a named User-agent group with Allow: / for the crawler you intend to permit, then confirm that a broader rule is not the only applicable policy. For ChatGPT search, OpenAI documents the relevant crawler name as OAI-SearchBot. OpenAI’s crawler documentation

Does allowing OAI-SearchBot allow model training?

OpenAI documents OAI-SearchBot for ChatGPT search and GPTBot separately for model-training-related crawling controls. Treat them as separate permissions and write separate rules if your policy differs. OpenAI’s crawler documentation

Can robots.txt keep private pages private?

Robots.txt cannot keep private pages private. It is a crawler preference protocol, not an authentication system, so private pages need access controls that require authorization. RFC 9309’s security considerations

Will unblocking a crawler guarantee AI-search visibility?

Permission to crawl is not a guarantee of retrieval, ranking, citation, or display, and the available crawler documentation does not promise those outcomes. OpenAI’s crawler documentation