Meta’s Scraping Clause Is Gone. Its Crawlers Are Not.

Meta added a clause to its commercial terms broad enough to cover training AI on content it crawls from business websites. Publishers noticed, lawyers piled on, and the provision disappeared before the terms took effect. The retreat is real, but it is narrower than it looks — and it changes nothing about what is already hitting your server.

“A Meta spokesperson insisted that the updated terms did not reflect a change in the way Meta crawls the web.”

Anthony Ha, TechCrunch

Our take

Read that quote twice, because it is the whole story. Per MediaNama’s reporting, the added language in Meta’s commercial terms said Meta “may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train” — and Meta first cut the word “train” and limited the permitted uses to advertising and business products, then removed the provision entirely. The terms were scheduled to take effect October 30, 2026. Meta’s own statement was that “the crawling terms we consolidated only applied to our ads and business products,” that the change “does not change how Meta uses web crawlers,” and that site owners can use robots.txt to signal their preferences.

So: the paperwork changed and the behavior did not. If a Meta crawler was fetching your pages last month, it is fetching them this month. Treat the withdrawal as a legal cleanup, not as a protection you now enjoy.

The detail most coverage skipped is where the clause lived. Commercial terms are the agreement you accept to run ads and use business products — Business Manager, lead forms, the pixel, catalog feeds. That is a consent surface, and it is not the open web. The practical reading is that Meta was seeking these rights from advertisers and business users regarding domains those users control, which is a very different thing from asserting a right against the whole internet. If you run Meta ads, you are inside that agreement. If you do not, you were never the party being asked — and your pages are exposed through ordinary crawling instead, which no clause was ever needed to permit.

That reframes the advice Meta offered. Pointing site owners at robots.txt sounds like a remedy; it is a courtesy signal. MediaNama quotes Benjamin Sobel of the University of Wisconsin making the point directly: robots.txt does not by itself create an enforceable right to prevent scraping. A well-behaved crawler obeys it. A crawler that does not obey it suffers no consequence from the file alone. Meanwhile James Grimmelmann of Cornell Law said the revised wording still covered AI training even after the first edit — which tells you how much interpretive room a sentence in a terms document carries, and how little comfort the narrowing provided.

Notice also who applied the pressure that worked. The European Publishers Council moved toward European regulatory scrutiny, the UK News Media Association called for government intervention, and the News/Media Alliance called the changes “outrageous and probably illegal.” That is organized publishing with regulators on speed dial. A plumbing company in Henderson has none of that leverage. The lesson for small and mid-sized businesses is unglamorous: the protection you get from this episode is whatever you implement on your own server, because nobody is lobbying on your behalf.

And expect the clause to come back in a narrower coat. The sequence — ship broad language, get caught in days, trim it, then withdraw it before the effective date — reads as a process failure, not a change of heart. Every major platform wants training rights to content flowing through its business products, and the commercial-terms update is the cheapest place to get them. Plan for the next version, not for the retreat.

For an agency operating a portfolio of client sites, the deliverable here is a written crawl policy you can apply the same way across every property: a maintained robots.txt, server-side or firewall rules for the crawlers you actually want to block, terms-of-use language on the site asserting your position, and a standing review whenever a platform updates its business terms. The reason to standardize it is that the next clause will not arrive with a press cycle attached.

What this means for your business

• Open your robots.txt today and see what is actually in it. Most small business sites have never been edited past whatever the theme or plugin wrote. Decide deliberately which AI and platform crawlers you allow, and write it down.
• Back the file with a rule the crawler cannot ignore. Add user-agent blocking at your host, CDN, or firewall for anything you genuinely want kept out. robots.txt is a request; a server rule is an answer.
• Pull your server logs for the last 30 days and list the top non-human agents. You cannot form a position on crawling until you know who is actually fetching your pages and how often.
• If you run Meta ads, read the commercial terms, not the news coverage. You are a party to that agreement. Have whoever handles your contracts skim it before October 30 and flag anything touching content rights.
• Put your crawl position in your own terms of use. A clear clause stating what automated collection you permit gives you a contract argument later. Robots.txt alone gives you none.

Does removing the clause stop Meta from crawling my site?

No. Meta said the change does not alter how it uses web crawlers. Blocking happens on your side, not in their terms document.

Is robots.txt legally binding?

Legal experts quoted in the reporting say it does not by itself create an enforceable right to prevent scraping. It signals a preference that well-behaved crawlers honor.

The clause history, Meta’s statements, and the publisher response are in the original report: Meta removes language that raised scraping and AI training concerns, by Anthony Ha for TechCrunch.

---