Brave’s unnamed crawler means you can’t block it without Google
Brave sells its search index of over 30 billion pages to AI companies via an API, and its crawler has no separate user agent, so robots.txt can't target it. You can only block it by also blocking Google, or by using noindex.
- Brave's crawler follows Googlebot rules and has no user agent of its own, so robots.txt cannot single it out.
- Your pages may feed AI answers via Brave's API without any guaranteed click back to you.
- Check your site with site:yourdomain.com on search.brave.com and watch the New York Times lawsuit.
Ask your own AI: ChatGPT Perplexity Claude AI Mode
On this page
Brave’s crawler does not advertise its own user agent, which means site owners cannot block it in robots.txt without also blocking Google. The detail surfaced in a column on Search Engine Journal (originally published on No Hacks), written after the New York Times sued Brave on November 2. This adds to our earlier report on the Times lawsuit against Brave.
What is in the Brave search index, and who buys it?
As reported by Search Engine Journal, Brave sells access to its index through the Brave Search API. Brave says the index holds over 30 billion pages and names Cohere, Mistral, Perplexity and You.com as customers. The Times’ complaint also lists Anthropic and xAI as suppliers-in-question; none of the AI companies is being sued, and Brave had not responded to the claims when the column was written, so they remain allegations.
Brave’s own help page says its crawler does not advertise a differentiated user agent. Its stated reason: some sites admit only Google, so a Brave-named bot would be turned away. Instead, Brave says it obeys whatever Googlebot is barred from reading.
Can you block Brave’s crawler in robots.txt?
Not selectively. Robots.txt works on named bots, and Brave has no name to target, so the choice is both Google and Brave, or neither. The help page points to the noindex tag as the exit, and that also tells Google to drop the page. The author, who runs a site with 13 AI bots listed in robots.txt, found 16 of his pages in Brave’s results, including a stored copy of a homepage description he had since replaced.
Why does the Brave search index matter for AI visibility?
This is interpretation, not a reported fact: an index sold to AI companies is a different deal from classic search. The complaint quotes Brave describing an “LLM Context” product that supplies extracted page chunks so no further scraping is needed. Whether an AI answer built from those chunks credits or links to you is the AI company’s decision. Meanwhile, the court outcome will not automatically bind other sites, but it may shape how crawler transparency is treated. Publishers face a related fight over hidden crawlers: News Corp sued Brave in July 2026, according to Semafor, as cited by the column.
How do I check if my site is in Brave’s index?
- Go to search.brave.com and search site:yourdomain.com. Anything listed is in the index Brave sells.
- Compare the stored descriptions with your live pages to see how fresh the copy is.
- Decide whether noindex on specific sections is worth losing Google visibility for. Usually it isn’t, so for most sites it is a policy question rather than a quick fix.
- Review your robots.txt strategy: it controls named AI crawlers (such as those from OpenAI and Anthropic) but not Brave-fed answers.
- Track referrals from AI assistants to see whether being in these indexes brings clicks back. Follow more in our AI Search & GEO coverage.