Meta Expands Its Right to Crawl Websites of Businesses Using Its Products
From October 30, Meta’s crawling rights will extend beyond advertisers to other businesses using its products. The terms also allow Meta to use collected content to develop and train its products.
Meta has updated its Commercial Terms. From October 30, the company will have permission to crawl websites associated with businesses that use its products and to use the content it collects, including to train its products. As Search Engine Journal reports, the change will affect not only advertisers but also businesses that, for example, run a Facebook page without buying ads.
Which Websites Are Covered
A new clause in the Commercial Terms allows Meta and those acting on its behalf to collect content using automated tools. It covers domains and pages associated with a business’s goods or services, as well as URLs made available through the business’s or its users’ use of Meta products. The terms also allow the creation of test credentials to access those pages.
Meta may use the content it collects to develop, improve, research and train its products. The updated Self-Serve Ad Terms, which also take effect on October 30, include the same permission to crawl. For advertisers, this right is not entirely new: the current ad terms already allow Meta to crawl their websites, including for research and training.
The main change is which businesses are covered. The Commercial Terms dated March 6, 2024, contain no crawling clause. The new version ties permission to any business use of Meta products, not just advertising. Meta considers continued use to mean acceptance of the updated terms; the terms tell those who disagree to stop using its products for business.
Meta spokesperson Andy Stone disputed the interpretation that the change concerns crawling news websites. He said the aim of the update is to obtain accurate information about products on third-party websites for ad and content recommendations. However, lawyers interviewed by Nieman Lab believe the wording could give Meta an argument for using publishers’ content to train AI. How a court would assess that argument remains an open question.
What We Know About Crawling Restrictions
Meta points to robots.txt as a way to control access to content. But neither the updated Commercial Terms nor the Self-Serve Ad Terms explain how robots.txt settings interact with the permission the company receives when the terms are accepted. Lawyers interviewed by Nieman Lab say Meta may be able to invoke the terms even when a site has instructions restricting crawling.
Meta’s documentation distinguishes crawlers by purpose. Meta-ExternalAgent is used, among other things, to train foundation AI models or improve products by indexing content. Meta-WebIndexer helps generate Meta AI search results: according to the company, allowing it access enables links to pages to appear in answers. Sites can set different rules for these crawlers—for example, blocking the first while allowing the second.
There are caveats. According to the documentation, Meta-ExternalFetcher may bypass robots.txt when retrieving a page at a user’s request. FacebookExternalHit, which generates link previews, may also bypass the file for safety or integrity checks. Meta did not answer whether test credentials could be used to access paywalled content.
What Website Owners Should Check
Before the terms take effect, businesses using Meta products should identify which of their domains and URLs are connected to those products. This applies not only to businesses running ads: a Facebook page or the use of Meta Business Suite also counts as business use.
The next step is to review robots.txt rules for individual Meta crawlers based on the site’s goals. The company describes access for indexing in Meta AI search and access for a crawler that may be used for training as separate scenarios. But robots.txt does not resolve the question raised by the new terms: they do not say whether the file limits the scope of permission to use content.
For publishers, the uncertainty goes further. The European Publishers Council is preparing to take the matter to the European Commission. It is concerned that Meta could interpret a user sharing a link to an article as grounds for using the article’s content. That is a concern, not an established consequence of the update.
My View
I wouldn’t reduce the decision to a single robots.txt setting. First, a business needs to understand which Meta products it uses and which pages are connected to them. Then it should check the rules for specific crawlers. Without that, it’s hard to assess what will actually change for the site after October 30.
I also wouldn’t present blocking as a universal answer. Both control over access and the opportunity to receive traffic from Meta AI search matter to a website. It’s more useful to treat these as separate scenarios and check the results against the data than to assume all Meta crawlers work the same way.
Sources
Where the news comes from. The text is a retelling in the author’s own words; the facts come from the source, the opinion is the author’s.
- Meta’s New Terms Expand Crawling Rights Beyond Advertisers via @sejournal, @MattGSouthern www.searchenginejournal.com
SEO specialist
I handle search analytics for several online stores and check changes against data, not search-result screenshots. In recent months, I’ve been keeping a close eye on how organic demand is changing.
All posts by the authorRelated articles
Google Adds Indexing Advice for Link-Sharing Pages to Its Spam Guidelines
Google has updated its spam guidelines with advice on indexing pages used for link sharing, collaboration, and online tools. If a page should not be found through search, Google recommends noindex.
Microsoft and Google Back a Shared Opt-Out for AI Training on Website Content
Microsoft and Google have backed a shared way for websites to opt out of AI training. Microsoft will also let site owners separately restrict the use of their content in Bing and Copilot answers.
Google, OpenAI, Microsoft and Anthropic Plan a Shared AI Training Opt-Out in robots.txt
Google, OpenAI, Microsoft and Anthropic plan to support a shared robots.txt marker so site owners can set one preference for AI training. Rollout is expected in 2027.
Discussion 5
Eva P.
For a small shop, I’d add this to the launch checklist: check whether product pages expose customer reviews, photos, or other user-submitted content, and make sure the site’s privacy and consent language still matches how that content may be reused. Less exciting than picking a new collection, but probably cheaper than untangling it later.
Stefan Evtimov
@eva.p, quick clarification: the updated terms sound broader than just reviews and customer photos—the crawl can cover pages associated with a business’s goods or services, and URLs exposed through its or its users’ use of Meta products. So would you treat this as a domain-wide crawl check too, not only a review of user-submitted content and consent wording?
Karolína Ondřej Post author
@stefan.ecom, yes—check the domain and relevant pages, not only user-submitted content. The terms cover pages associated with a business’s goods or services and URLs surfaced through use of Meta products, though they do not specify exactly how that crawl is scoped.
Filip Fiala
The clause about creating test credentials seems especially important for businesses with booking or account areas. Does the updated wording explain how Meta is expected to access those pages, or what practical controls a site owner can use to prevent automated access without disrupting normal sharing through Facebook?
Karolína Ondřej Post author
The terms permit automated collection and test credentials, but they do not specify an access method or a dedicated opt-out for these pages. Owners can review their robots.txt and access controls, though blocking crawlers may not be the same as preventing content from being shared through Facebook.