How AI search reads your business website, and how to check it

When people ask ChatGPT or Perplexity for a bike repair shop in Bern or a dentist who takes new patients, the answer can name a few businesses and link to their websites. Nobody outside these companies knows exactly how those names are chosen. One part is in your hands, though: whether their crawlers can read your website at all.

This guide explains what AI crawlers see on a small business website, how to check yours in a few minutes, and what to fix first. frascati, our website and app builder, handles some of it for you; we say which parts, and which parts stay with you.

How AI search finds your website

An AI assistant answers from what its model learned and, for current questions, from pages it looks up. To look things up, each company runs crawlers that read websites, much like a search engine does. OpenAI lists its crawlers on one page, and three of them matter here: OAI-SearchBot reads pages for search in ChatGPT, GPTBot collects content that may be used to train OpenAI's models, and ChatGPT-User may visit a page while ChatGPT answers someone's question. Anthropic and Perplexity run crawlers of their own. Google's AI Overviews link only to pages that are indexed in Google Search, which reads pages with Googlebot.

So the first rule is simple: do not lock out the crawlers of the assistants you want to appear in.

What a crawler sees

Many websites today are put together in the visitor's browser: the server sends an almost empty page and a script that fills it in. Google runs that script before it reads the page, as its Search Central documentation describes. Many AI crawlers do not run it.

For a crawler that does not run scripts, a website built only in the browser can be a title and an empty page: no services, no prices, no opening hours. It cannot quote what it cannot read.

Check your website in a few minutes

• Open your website on a computer, right-click an empty spot and choose View Page Source. In Safari, first turn on Show features for web developers under Safari, Settings, Advanced. What you see is what a crawler gets before any script runs.
• Search that source for a sentence from your page, like your opening hours or one of your services. If you find it, crawlers that do not run scripts can read it. If you find only code, they cannot.
• Open your address with /robots.txt at the end. A line Disallow: / under User-agent: * or under a crawler's name, such as OAI-SearchBot or PerplexityBot, keeps that crawler out.
• Open your address with /sitemap.xml at the end. It should list every page you want to be found.
• Ask Google and an AI assistant about your business by name and town. Write down what they get wrong or leave out. That list is where to start.

What to fix first

• Put the facts on the page as text: your services, prices, opening hours, address and phone number. Text inside photos or behind a script is easy to miss.
• Give each important service a page of its own, with a heading that says what it is and where. A clear page is easier to quote than one long page about everything.
• Use the same name, address and phone number everywhere: on your website, in your Google Business Profile and in directories.
• Answer the questions customers ask before they call: what it costs, how long it takes, whether there is parking, how to book.
• Give every page a title and a description that say what the page is about and where.
• Let the crawlers in, unless you have a reason to keep one out.

None of this guarantees a mention. Anyone who promises you a place in ChatGPT's answers is guessing. What you can make sure of is that when an assistant looks, it finds the right facts.

How frascati handles it

• Right after you publish, frascati saves a finished HTML copy of every page. Crawlers that do not run scripts read that copy, with all the text, headings and links, and visitors get the live website.
• At a frascati address, like frascati.app/p/your-name, crawlers read your home page. On your own domain, with Pro and Business, every page has its own address, and your website gets its own sitemap, robots.txt and llms.txt. Its robots.txt lets every crawler in.
• On your own domain, frascati tells Bing about new pages right after you publish.
• The SEO view in the editor checks every page the way a search engine reads it: titles, descriptions, headings, broken links, placeholders left, your contact details and the business details Google reads for local searches. It uses no credits.
• frascati does not make up facts about your business. What it does not know appears as a visible [placeholder] until you fill it in, or is left out, so your website does not show invented prices or opening hours.

What stays with you: the facts, a Google Business Profile with your website's address, and keeping both current when something changes.

Frequently asked questions

Should I block AI crawlers?

That is up to you. Blocking a training crawler such as GPTBot tells OpenAI not to use your text to train its models. Blocking a search crawler such as OAI-SearchBot keeps your website out of ChatGPT's search answers, though it can still appear as a plain link. A business that wants to be found usually leaves the search crawlers in.

Do I need an llms.txt file?

No. llms.txt is a proposal for a plain-text summary of a website. Crawlers read your pages without it, so the text on the pages matters more. It does no harm to have one, and frascati writes one for websites on their own domain.

Does Google read websites built with JavaScript?

Yes. Google runs the scripts before it reads a page, though that can take longer than for plain HTML. Other crawlers may not run them at all, which is why the text of your pages belongs in the HTML.

How soon will an assistant know about my new website?

Nobody can say. Crawlers come back on their own schedule. OpenAI says a change to robots.txt can take about a day to reach ChatGPT's search. A sitemap, and links from your Google Business Profile and from directories, help crawlers find new pages.