Brave Sells AI Companies Access to Its Search Index: What It Means for Websites
Brave gives AI companies access to a search index containing more than 30 billion pages. Here’s how to check your site and why robots.txt can’t block Brave’s crawler separately.
Brave sells access to its search index through the Brave Search API; the company names Cohere, Mistral, and Perplexity among its users. On November 2, the New York Times sued Brave, challenging its use of the newspaper’s content. As Search Engine Journal reports, this affects more than publishers: pages from ordinary websites can also end up in the index that AI services draw on for data. You can check your domain in Brave Search with a site:yourdomain query.
How AI services use Brave’s index
When an AI assistant needs up-to-date information from the web, it can query an existing search index rather than crawl websites itself. Brave provides this access through an API. According to the company, its index contains more than 30 billion pages, with millions of new ones added every day. Brave also names You.com among its API users.
For a website owner, a conventional search result and an AI-generated answer are two different things. In search results, someone sees a page and can click through to it. An AI service can retrieve information from an index and put together an answer, but whether it directs the reader to the source is up to the service. The lawsuit describes a Brave product for language models that supplies page excerpts and source information in a format prepared for the model. The recipient does not need to crawl the website again.
The New York Times lawsuit also names Anthropic and xAI among the companies that, according to the plaintiff, receive data from Brave. The AI companies themselves are not named as defendants in this case. The situation with Claude is less clear: the article cites evidence from 2023 and March 2025, but its author could not confirm whether the service currently uses Brave’s index.
Why you can’t block Brave separately in robots.txt
In robots.txt, a website owner sets rules for crawlers by name. But according to Brave’s documentation, its crawler does not identify itself with a distinct user-agent. The company says this is because some websites allow Googlebot through while blocking other crawlers. Brave says it follows the rules for Googlebot: if Googlebot is barred from reading a page, its crawler should not read it either.
That means you cannot allow Googlebot to crawl your site while blocking Brave’s crawler with a separate robots.txt rule. To remove a page from its index, Brave suggests using noindex, but that will also remove the page from Google’s index. If a site relies on search traffic, this is not a targeted “don’t show it to AI” setting.
The New York Times alleges that Brave copied more than 30 billion pages without identifying its crawler separately or giving websites a way to block that crawler specifically. That is the plaintiff’s position, not a fact established by a court. Brave had not responded to the lawsuit at the time the article was published. On the same day, Daily News, Chicago Tribune, and Orlando Sentinel filed a separate case against the company.
What website owners should check
Start with a site:yourdomain query in Brave Search: it will show your domain’s pages in the available search results. The article’s author checked his site on November 3 and found 16 pages on the first page of results, including 13 articles. The homepage description was still the old one, even though he had changed it earlier that day. So the results may show stored data rather than the page as it appears right now.
This query can help you see which pages appear in Brave Search, but it cannot tell you whether a particular AI service used them or sent visitors your way. Being in the index does not mean people click through to your site. Developers and marketers should keep those questions separate before changing access rules: robots.txt cannot block Brave separately in the way described, while noindex also affects ordinary visibility in Google.
My take: think about the customer before fighting the index
I’d start by checking specific pages, not by solemnly declaring war on crawlers. I have a small shop, and people still search for specific products. If a customer is looking for an item, I want them to find its page and be able to place an order. Removing it from Google along with Brave just to tick a box on a task list is an idea I’d discuss with customers first. A presentation about total control over the index won’t place an order for them.
But I wouldn’t treat an AI-generated answer as equivalent to a conventional search result, either. In search, a page has a clear route to a visitor; when an excerpt from it appears in a ready-made answer, that route depends on another service. I’d keep an eye on which pages appear in search results and whether people click through, without treating the allegations in the lawsuit as proven facts. It won’t make the third redesign of my website any more urgent.
Sources
Where the news comes from. The text is a retelling in the author’s own words; the facts come from the source, the opinion is the author’s.
- Brave Sells Its Search Index To AI, And Your Website Is Probably In It via @sejournal, @slobodanmanic www.searchenginejournal.com
Georgian food shop owner
I sell products from my small shop and am growing online orders. The website has already been through two redesigns; I’ll start a third only after talking to customers, not after another trendy presentation.
All posts by the authorRelated articles
Google Adds Indexing Advice for Link-Sharing Pages to Its Spam Guidelines
Google has updated its spam guidelines with advice on indexing pages used for link sharing, collaboration, and online tools. If a page should not be found through search, Google recommends noindex.
Google, OpenAI, Microsoft and Anthropic Plan a Shared AI Training Opt-Out in robots.txt
Google, OpenAI, Microsoft and Anthropic plan to support a shared robots.txt marker so site owners can set one preference for AI training. Rollout is expected in 2027.
Microsoft and Google Back a Shared Opt-Out for AI Training on Website Content
Microsoft and Google have backed a shared way for websites to opt out of AI training. Microsoft will also let site owners separately restrict the use of their content in Bing and Copilot answers.
Discussion 4
Mārtiņš Kalniņš
For our home-goods catalog, I’ve found it more useful to check whether Brave surfaces the right product details than to treat every crawler as an emergency. If an AI answer gets materials or care instructions wrong, that’s a page-content problem worth fixing; blocking access might be the pricier kind of quick fix.
Nigar Yusifli
@martins.k, checking whether Brave surfaces the right details is useful, but I don’t think that makes access a secondary concern. Once product pages are available through an API, their content can be reused in contexts where you cannot inspect or correct the answer—and the article notes that robots.txt cannot block Brave’s crawler separately. I’d treat this as two distinct checks: make the source pages accurate, then decide whether that distribution is acceptable, rather than assuming better copy solves the exposure question.
Ivan Uzunov
That distinction makes sense. Since robots.txt can’t block Brave’s crawler separately, what practical signal would you use to decide whether the distribution risk outweighs the visibility benefit—especially for product details we can’t correct once they’re reused in an AI answer?
Jaakob Pärn
For a small store, I’d compare Brave referrals and orders with the cost of inaccurate product claims; if the index brings no measurable sales, I’d avoid exposing details that are hard to correct downstream.