Configure storefront robots.txt with AI
Plan crawl rules for a Hydrogen storefront using its actual routes. Check the sitemap, faceted collection URLs and the public response served to crawlers.
Configure storefront robots.txt with AI when a collection can generate dozens of URLs for the same catalogue. A single colour filter might be a useful landing page, while multiple filters combined with sorting may produce little more than repeated listings.
The hydrogen-robots-txt skill explains how to serve a public text response for the storefront and adapt rules to its real paths. It covers user-agent groups, the sitemap address and headers that influence caching. The result needs testing on the host that actually serves the shop.
What it does
- Identify crawlable product and collection pages
- Review search, cart and account route patterns
- Test single and combined filters against the proposed rules
- Point crawlers towards the correct sitemap
- Check content type, status and cache behaviour
Install and use
- Install the skill in a compatible coding agent and provide the Hydrogen app's framework, route list and sitemap location. Use a local server or an authorised test host to inspect the response.
- Ask, “Keep our single-colour collection pages crawlable. Review sorted URLs and multi-filter combinations against these examples.”
- Fetch the public robots file, inspect its headers and compare the rules with the pages the merchandising team wants customers to find.
FAQ
Source
Made by Shopify. Original skill: hydrogen-robots-txt, published on GitHub under the MIT licence.
View on GitHub