Reddit has updated its robots.txt file, preventing Bing and many other search engines from crawling the site. “Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site,” a Microsoft representative told Search Engine Land.
What happened. On July 1, 2024, Reddit updated its robots.txt file preventing many search engines and AI tools from crawling the site. Reddit did not prevent Google from crawling the site, despite what some people may have thought earlier this month. But Reddit did prevent most other crawlers from crawling the site.
Earlier this morning, Mark Williams-Cook notified me that Reddit results were dropping out of the Bing Search index. Then several media outlets began covering the news. I wanted to confirm that Bing’s crawlers were indeed blocked, because Reddit was using IP detection to show search engines one version of its robots.txt file and humans another version of the robots.txt file – as I explained earlier this month.
Bing has thus stopped crawling new content on Reddit, which is why when you filter Reddit results in Bing Search for the last week, you see nothing:
Microsoft confirmed. A Microsoft spokesperson told Search Engine Land:
“Microsoft respects the robots.txt standard and we honor the directions provided by websites that do not want content on their pages to be used with our generative AI models. Bing stopped crawling Reddit after they implemented their updated robots.txt file on July 1, which prohibits all crawling of their site.”
Reddit statement. Reddit spokesperson Tim Rathschmidt says in a statement to The Verge:
“This is not at all related to our recent partnership with Google. We have been in discussions with multiple search engines. We have been unable to reach agreements with all of them, since some are unable or unwilling to make enforceable promises regarding their use of Reddit content, including their use for AI.”
Why we care. With Reddit securing a licensing deal with Google, Reddit is able to play hardball with other search engines and AI tools. So Reddit has blocked most other search engines from crawling its content. Meanwhile, Google is driving insane traffic to Reddit these days, including testing special treatment in its search results for them.
It makes you wonder if other large websites can try to go down this route and where that might leave smaller publishers and content producers.
Meanwhile, don’t expect to see much new Reddit content being surfaced on Bing in the near future.
