All products

Configure storefront robots.txt with AI

Plan crawl rules for a Hydrogen storefront using its actual routes. Check the sitemap, faceted collection URLs and the public response served to crawlers.

Configure storefront robots.txt with AI when a collection can generate dozens of URLs for the same catalogue. A single colour filter might be a useful landing page, while multiple filters combined with sorting may produce little more than repeated listings.

The hydrogen-robots-txt skill explains how to serve a public text response for the storefront and adapt rules to its real paths. It covers user-agent groups, the sitemap address and headers that influence caching. The result needs testing on the host that actually serves the shop.

What it does

  • Identify crawlable product and collection pages
  • Review search, cart and account route patterns
  • Test single and combined filters against the proposed rules
  • Point crawlers towards the correct sitemap
  • Check content type, status and cache behaviour

Install and use

  1. Install the skill in a compatible coding agent and provide the Hydrogen app's framework, route list and sitemap location. Use a local server or an authorised test host to inspect the response.
  2. Ask, “Keep our single-colour collection pages crawlable. Review sorted URLs and multi-filter combinations against these examples.”
  3. Fetch the public robots file, inspect its headers and compare the rules with the pages the merchandising team wants customers to find.

FAQ

Source

Made by Shopify. Original skill: hydrogen-robots-txt, published on GitHub under the MIT licence.

View on GitHub