As of September 29, X has updated its terms of service to explicitly prohibit data scraping and crawling without prior written consent. The update marks a significant change from the platform's previous stance, which permitted crawling in accordance with its robots.txt file.
Interestingly, X's robots.txt file has been altered to only permit Google's robot crawlers, excluding all others. This change comes following an existing "firehose deal" between Twitter and Google, which allows tweets to surface in search results. However, the status of this deal under X's new management remains unclear.
Elon Musk, owner of X, has been vocally against data scraping for AI training. In April, Musk threatened to sue Microsoft for using Twitter data for this purpose, and in July, he initiated a lawsuit against unidentified entities for scraping data. This move appears to be in line with these earlier positions.
In addition to these changes, X has also updated its privacy policy, stating the potential use of public data to train AI models. This is in line with Musk's recent founding of xAI, a company intended to leverage public data for AI training.
The new terms also imply increased data collection, including biometric data and information on education and job history. This broadening of data usage indicates a shift in how X views and utilizes its user data.
Elon Musk’s X is taking a more protective stance on its user data, most notably through the recent ban on scraping and crawling. It will be interesting to see how these changes impact user privacy and the overall data ecosystem surrounding X.
