Meta Removes Controversial Scraping Language from Terms After Backlash

Meta has updated its commercial terms to remove language that experts interpreted as granting the company the right to scrape content from websites, including news sites, for AI training. The change follows backlash from publishers and legal scholars who argued the clause could override website owners' preferences expressed through robots.txt files.

The removed language stated that Meta "may use any content from such domains, web pages, or URLs to provide, develop, improve, research, train, and maintain the integrity of, the Meta Company Products, services, and features." As reported by The Verge, the clause was widely criticized, with Neiman Lab and James Grimmelmann, a professor of digital and information law at Cornell, among those interpreting it as an attempt to secure free access to scraped content for AI development and to bypass robots.txt blocking instructions.

Meta initially pushed back. Andy Stone, Meta's vice president of communications, and spokesperson Daniel Roberts said that "There is no change to how Meta uses web crawlers; any suggestion to the contrary is false." However, the company has now removed the controversial language entirely. In a follow-up statement to The Verge, Roberts said: "We've heard you and want to clarify: the crawling terms we consolidated only applied to our ads and business products, and we've clarified that in the terms themselves. It does not change how Meta uses web crawlers. As always, content owners can use robots.txt to signal their preferences. This is not about scraping news sites, and these updates don't give us new rights that supersede those preferences."

The revision appears to address concerns that the terms could be read as overriding robots.txt signals. Meta maintains that its crawling practices remain unchanged and that the update merely clarifies the scope of the terms. The company emphasizes that content owners can still use robots.txt to express their preferences, and that the changes do not grant new rights that supersede those preferences.

This incident highlights the growing tension between AI developers and content creators over the use of web data for training machine learning models. As AI companies seek vast amounts of data to improve their systems, publishers and website owners are increasingly vigilant about how their content is used and whether crawlers respect their instructions. Meta's decision to remove the language may be seen as a concession to those concerns, though the underlying debate over AI training data and web scraping is likely to continue.

For now, the removal of the clause suggests that Meta is sensitive to the criticism and willing to adjust its terms in response. The company's clarification that the terms were intended only for its ads and business products may reassure some, but the episode underscores the scrutiny that big tech faces over its data practices, especially as AI becomes more central to their offerings.

The Verge reported that the update was made after feedback from stakeholders. Meta's move could set a precedent for how other companies craft their terms of service regarding AI training and web crawling. It remains to be seen whether other firms will follow suit or face similar pushback.

This story was originally reported by The Verge.