Your post/comment has been removed because your account has a low CQS Score.
Please contribute more positively on Reddit overall before posting. Cheers 😀
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/SEO) if you have any questions or concerns.*
you’ve hit on the big secret: you’re right – robots.txt is there to block bots from crawling or indexing pages, it doesn’t “optimize” the site in anyway.
Reddit ranks from lots of google search activity, backlinks, internal links, page title relevance and because (mostly) Google promoted it so much around the time of the HCU
ergo, robots.txt is not about SEO except that it falls into the SEO toolset for excluding bot access or index access.
And thats why boiling SEO down to good/bad to-do checklists is dumb because its a system.
Reddit has a data licensing deal with Google (announced early 2024), and since mid-2024 it blocks other crawlers like Bing and most AI bots at the robots.txt level. What you see is the rule for everyone else, Google is allowed through under the agreement. Also worth knowing that robots.txt only stops crawling, not indexing, so Google can list a URL it hasn’t crawled if enough links point to it.
They have an agreement with Google. Google has a direct feed to Reddit. They don’t need to crawl the site.
Your post/comment has been removed because your account has a low CQS Score.
Please contribute more positively on Reddit overall before posting. Cheers 😀
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/SEO) if you have any questions or concerns.*
This is for other search engines. They have an exclusive agreement with Google so their robots.txt file does not apply to Google.
hey u/Soft_Maintenance_326
you’ve hit on the big secret: you’re right – robots.txt is there to block bots from crawling or indexing pages, it doesn’t “optimize” the site in anyway.
Reddit ranks from lots of google search activity, backlinks, internal links, page title relevance and because (mostly) Google promoted it so much around the time of the HCU
ergo, robots.txt is not about SEO except that it falls into the SEO toolset for excluding bot access or index access.
And thats why boiling SEO down to good/bad to-do checklists is dumb because its a system.
Reddit has a data licensing deal with Google (announced early 2024), and since mid-2024 it blocks other crawlers like Bing and most AI bots at the robots.txt level. What you see is the rule for everyone else, Google is allowed through under the agreement. Also worth knowing that robots.txt only stops crawling, not indexing, so Google can list a URL it hasn’t crawled if enough links point to it.