Being found
robots.txt
The file that tells crawlers where they may go.
What is robots.txt?
robots.txt is a small text file at the root of a website that tells search engines and other automated visitors which parts of the website they may read. It can point to the sitemap, and it can, by accident, block a whole website from Google.
What it means for a small service business
The classic disaster is a developer blocking everything while a website is being built and forgetting to unblock it at launch. The newer mistake is blocking the AI crawlers, which means the assistants your customers ask can never read you.
Does your business need to do anything about robots.txt?
Check it once. It should allow Google and the AI crawlers and name the sitemap. Sydex builds block everything during the pitch and open up at go-live, on a checklist.
See also
Act on it
The tools and guides that deal with robots.txt on this website, then the free review if you want it looked at properly.
Run the one-minute health check →Get your free review