OpenAI says robots.txt may not apply to the ChatGPT fetch bot
Per a Search Engine Journal report dated August 14, 2026, OpenAI states that robots.txt rules may not apply to the ChatGPT-User bot. The reasoning: when a user asks a question, the resulting page request counts as a direct user action rather than a decision made by a crawler.
The distinction between crawling and a user request
Per the report, OpenAI argues that user initiated requests differ from automated crawling, and therefore that robots.txt rules may not apply to ChatGPT-User.
Per the company documentation, this bot visits pages when ChatGPT users ask questions. The page request originates from a person question at that moment rather than from a crawling program schedule.
Other agents are named in the report as well. OAI-SearchBot determines visibility in ChatGPT search results. Bytespider, Youbot, Claude-User and Perplexity-User are also mentioned.
The compliance picture and the cost of blocking
Per the report, ChatGPT-User reached disallowed pages on more sites than any competing AI bot.
There is a regional breakdown too. On European sites roughly 15 percent of identified AI fetchers accessed disallowed URLs. ChatGPT-User is disallowed by 26 percent of North American sites, yet appears more aggressively in the European data.
The critical distinction is stated plainly: disallow directives represent requests only. robots.txt is not a technical wall but a courtesy rule expected to be honored.
Blocking carries a cost as well. Per the report, sites blocking both ChatGPT-User and OAI-SearchBot lose the visibility benefit while their retention of fetch control may prove ineffective. The report also notes that Cloudflare changes in September 2026 will shift enforcement to the network layer rather than relying on crawler compliance.
The gap between a courtesy rule and technical control
The real point here is this: robots.txt was never a security mechanism, it was an agreement. Search engines honored that agreement for many years, and site owners began treating it as a control. When AI clients opened the scope of the agreement to debate, the true nature of the tool became visible. That assessment is ours.
Second, the logic of the user request rationale. When a person visits your site with a browser you do not consult robots.txt, because that request is already personal. The OpenAI framing extends that to the agent. The logic looks consistent, but the consequence is that any client acting on a user behalf can use the same rationale.
Third, blocking cuts both ways. Closing a site to AI clients protects the content and closes off visibility at the same time. That decision needs measurement rather than instinct, and the new Clarity scrape to referral ratio offers exactly that measurement. That connection is our note.
Fourth, enforcement moving to the network layer. As the report notes, changes on the Cloudflare side take the decision out of client goodwill and into infrastructure. For a site that genuinely wants to block, the right place is network rules rather than robots.txt.
What it means for businesses in Türkiye
The assessment below is not in the sources, it is our reading. The report contains no Türkiye specific breakdown.
The robots.txt files of corporate sites here were mostly set up years ago and never reopened. Assuming that a rule written in that file stops AI clients is no longer a safe assumption.
For businesses producing content the decision runs both ways. If you invest in blog and guide material you may want to appear in AI interfaces, in which case blocking works against you. If you produce original research, datasets or paid content you may want protection. Those are different decisions and one file does not settle them.
Three practical steps. First, open your robots.txt and read deliberately what it says to which AI clients. Second, if there is something you genuinely want to block, enforce it at the network or server layer, because robots.txt alone is not enough. Third, measure whether the clients you block send you visits, because a blocking decision you do not measure cannot show you the visibility you lost.
The UNALSOFT take
On the web design and agentic AI side we always explain this in the same order: clarify what you want first, then enforce it at the right layer. If you want visibility, closing the door hurts you. If you want protection, robots.txt is not enough, because it is a file of requests. The mistake we see most often is being stuck between the two: writing a block in the file, believing you are protected, and being surprised at not appearing in AI interfaces. Both outcomes come from the same decision, and when that decision is not made deliberately you lose on both sides.
Sources
Search Engine Journal, robots.txt and the ChatGPT fetch bot · OpenAI developer documentation, bots
What does your robots.txt say?
Let us clarify your strategy toward AI clients together.