Shopify Squad About

Shopify Squad / Selling through AI chats

Block or allow AI crawlers in Shopify's robots.txt

Edit robots.txt.liquid for GPTBot, ClaudeBot and others, and know what it does not stop.

4 min read

To block or allow an AI crawler on Shopify, add a robots.txt.liquid template to your theme and write a User-agent group for that crawler below Shopify’s default rules. It changes what crawlers on the open web are asked to do. It does not stop Shopify Catalog from sending your products to ChatGPT, Copilot or any other AI channel you have left on.

What robots.txt can and can’t do on Shopify

  • Bot management is already on. Shopify handles it at the network layer for every store, with no setting for you to change, and doesn’t recommend putting a proxy in front of Shopify.
  • robots.txt is a request. Shopify calls the rules “directional and advisory”. A crawler can ignore them.
  • Blocking only affects the open web. Products still go out through Catalog to the agentic channels that are switched on. To stop that, turn the channel off instead: see Shopify agentic storefronts opt out.

Decide which crawlers to block

Most AI companies run separate bots for model training, for their search features, and for fetching a page when a user asks.

CrawlerCompanyWhat the company says it’s for
GPTBotOpenAIContent that may be used to train its foundation models
OAI-SearchBotOpenAIShowing sites in ChatGPT search answers
ChatGPT-UserOpenAIVisiting a page when a user asks ChatGPT; robots.txt rules may not apply
ClaudeBotAnthropicCollecting web content that could contribute to model training
Claude-SearchBotAnthropicImproving search results for users
Claude-UserAnthropicFetching pages to answer Claude users’ questions
Google-ExtendedGoogleUse of your content for Gemini training and grounding; it doesn’t affect Google Search inclusion or ranking
PerplexityBotPerplexityShowing and linking sites in Perplexity search results; not used for training
Perplexity-UserPerplexityVisiting pages for user requests; generally ignores robots.txt

A common setup is to block the training bots (GPTBot, ClaudeBot, Google-Extended) and leave the search bots alone, so your store can still be cited in AI search answers. OpenAI states that its settings are independent: you can allow OAI-SearchBot and disallow GPTBot. Blocking OAI-SearchBot removes your site from ChatGPT search answers, although it can still appear as a navigational link.

Read Shopify’s warnings first

  • Editing the file is an unsupported customization, and incorrect use can lose you all of your traffic.
  • Shopify Support can’t help with edits to this file. Shopify suggests a Shopify Partner if you need help.
  • Remove earlier workarounds first, including rules set through third-party services such as Cloudflare.

Every Shopify store already has a default robots.txt that allows products, collections and pages and blocks admin, cart, checkout and account pages. You are adding to it, not replacing it.

Create robots.txt.liquid

  1. In your Shopify admin, go to Online Store.
  2. Next to your theme, click the horizontal menu icon and select Edit code.
  3. Click Add a new template, then select robots.
  4. Click Create template.
  5. Add your rules (next section), then save the changes in your published theme.

Add a rule for one crawler

The new template starts with Liquid that prints Shopify’s default rules. Keep it. Add groups for other crawlers by hand, below that Liquid. To block three training bots, your file looks like this:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

To explicitly allow a crawler instead, use Allow: / in its group. Shopify advises against replacing the template with plain-text rules, because Shopify’s defaults can change and a static copy goes out of date.

Check that it worked

Open https://yourstore.com/robots.txt and look for your new groups at the bottom. Shopify says changes take effect immediately, but crawlers may take time to react. OpenAI says it can take about 24 hours for its search systems to adjust after you change robots.txt.

Undo your changes

  1. Go to Online Store and select Edit code for the theme.
  2. Right-click robots.txt.liquid and choose Delete permanently.
  3. Click Delete.

Your store goes back to Shopify’s default rules.

  • agents.md and llms.txt are separate from robots.txt. They describe your store to agents instead of granting or refusing access. See Shopify llms.txt.
  • Single products can be kept out of every AI channel by setting them to Unlisted, which also hides them from search engines and store search.
  • Products missing from AI chats aren’t a robots.txt problem, since robots.txt doesn’t affect Catalog. Check Shopify Catalog ineligible.

Sources