Can AI Read Your Site?
Before an assistant can recommend your hotel, it has to be able to reach and read your website. Plenty of hotels are blocking the crawlers that matter without knowing it, sometimes through a line in a file, sometimes through a security setting nobody remembers switching on.
Enter your address and we will check three things: what your robots.txt allows, whether anything turns crawlers away, and whether your content is in the page source at all.
Important note about what this tool can and cannot check · read more ↓
This tool acts as an independent crawler. It asks your site the way AI crawlers do, using their published names, and reads the same public files they read.
Every website and server setup is different. Some sites, particularly large platforms and anything behind enterprise bot management, refuse or slow requests from checkers like ours. When that happens we cannot get a full or reliable result.
Where that happens, we say so in the results, explain what failed, and point you to what you can check yourself. If we cannot read your robots.txt, you can open it in your browser, paste it in, and we will read the rules from that instead.
About this toolWhat this check is, and what it is not · read more ↓
What we check
- Your robots.txt, read the way the standard says crawlers should read it.
- Whether your site serves a page to a request carrying a crawler’s name.
- Whether your content is in the HTML your server sends, or only appears once JavaScript has run.
Why the firewall test is only an indication
A crawler’s name is just text that any software can send. Real crawlers also prove who they are, by arriving from published addresses, and we cannot copy that. So a refusal here means something filters requests carrying that name, which is the most common way hotels block AI by accident. It does not prove the real crawler is blocked. It runs the other way too: a site can filter by address and ignore the name, in which case we get through and the real crawler does not.
How we treat your site
A handful of spaced-out requests to your homepage, plus your robots.txt and sitemap. Every request names this tool in a header, so you can identify us in your logs. If we meet a security challenge we record it and stop, never try to get past it. Results are kept a few hours so a second look costs your server nothing.
Sites we cannot check
The big hotel groups run enterprise security systems we cannot measure from the outside. Hilton, Marriott, Wyndham and Four Seasons all do. When we meet one we name it and say plainly that you will need to check it yourself, rather than guess. Your robots.txt results still stand. Independent hotels rarely run these systems, including those behind Cloudflare.
Worth knowing that Cloudflare blocks AI crawlers at the edge rather than through robots.txt, so that setting never appears in a robots.txt file. You will find it under Security Settings in your Cloudflare dashboard. Cloudflare’s documentation explains it here.
What it does not tell you
Whether assistants actually mention your hotel, or what they say when they do. This checks whether the door is open, not what happens once someone walks in.