Anyone who runs a business Page on Facebook and keeps using Meta's products will, from 30 October, allow Meta to crawl and download websites associated with their products or services. Following criticism, Meta has pointed publishers to the robots.txt file, which lets a website switch off its crawler, but the terms that take effect on 30 October do not commit Meta to respecting it. On Friday 16 October 2026, the word train was dropped from the list of purposes for which Meta may use content from these websites.
The word train was still in the terms when Stephen Beale noticed the announcement about them in Meta Business Suite. Beale is the founder and editor of the website The Steampunk Explorer. He read the original wording to mean that, in the case of Facebook business users, Meta has “the right to go out and grab information from basically wherever you have a presence”. Beale seriously considered closing his accounts before 30 October.
Robots.txt is what remains to stop website crawling
Under the new terms, closing your accounts or otherwise ending commercial use of Meta's products is the only option for anyone who does not agree with them. Those who stay also give Meta access to content made available through the use of its products. Meta may regularly download updates from the websites. Software spiders, robots and test login credentials have not disappeared from the terms. Until now, website crawling was covered by the generative AI terms for advertising, specifically for URLs that a business itself provides to Meta for advertising.
Crawling of your own website is stopped by a disallow line for the relevant crawler in the robots.txt file. This is also how Meta-ExternalAgent is blocked. Among other things, this crawler visits websites to train foundation AI models. US law professor Benjamin Sobel compared robots.txt to a “Don’t look at my house” sign. In his view, robots.txt expresses a website's wish, which can be ignored unless it is legally enforceable.
A website's wish expressed in machine-readable form is provided for in the EU Directive on copyright in the Digital Single Market. The text and data mining exception in Article 4 does not apply where rightholders have expressly reserved such use in an appropriate manner, such as machine-readable means in the case of online content. Section 39c of the Czech Copyright Act provides for the same reservation for online content. As of 21 October 2026, Czech Radio's news website iROZHLAS has a Disallow line with a slash for the meta-externalagent crawler in its robots.txt file, which bars crawling of the entire site.
Meta may use downloaded content to develop products
The new wording allows Meta to use content from websites its crawler visits to provide, develop, improve, research and maintain advertising and business products. In a statement on Friday, Meta said the crawling rules applied only to its advertising and business products and that it had clarified this in the terms.
However, the term business product does not appear to be defined, according to Matt Topic, a US media and intellectual property attorney. In his view, the verb develop is “broad enough to include ‘train’”. James Grimmelmann, a professor of digital and information law, pointed out that the revision does nothing to restrict the phrase “associated with your products or services”, which would allow Meta to crawl entire websites.
Even before the revision, the European Publishers Council, an organisation of European publishers, formally appealed to two Executive Vice-Presidents of the European Commission over content from publishers' websites. According to its executive director, Angela Mills Wade, a business account or a link shared by a reader “should never be treated as a blanket licence to exploit a publisher’s content”. In Friday's statement, Meta also claimed that the changes do not give it new rights that would override robots.txt preferences.
According to the documentation, the Meta-ExternalFetcher crawler can bypass robots.txt preferences, because it fetches links at a user's request. The list of purposes from which train was dropped gained the word security, and the FacebookExternalHit crawler can bypass robots.txt during security or integrity checks.
Comments