Webflow’un Robots.txt bölümünde aşağıdaki bilgileri içeren ücretsiz bir giriş alanı vardır:
Kullanıcı aracısı:
İzin vermek: /
Site haritası: web sitem/sitemap.xml
Bu bölümü güncellemem ve belirli AI oyuncularını eklemem gerekli mi:
Kullanıcı aracısı: GPTBot
İzin vermek: /
Kullanıcı aracısı: CCBot
İzin vermek: /
Kullanıcı aracısı: ChatGPT-Kullanıcı
İzin vermek: /
Kullanıcı aracısı: OAI-SearchBot
İzin vermek: /
Ayrıca bu bilgilerden herhangi biri AYRICA bir llms.txt dosyasına da giriyor mu?
>Also, does any of this information ALSO go in an llms.txt file?
You’re kind of answering your own question: if llms was so important, wouldnt there need to be some reason why?
No, LLMs.txt is not doing aynthing. Why? Because LLMs are not independent search engines. They do not train off the whole WWW and they do not “remember” the whole www.
What they’re good at doing – esp ChatGPT – is pretending they are answering you but what they really do is go to Bing or Google, change the prompt via the QFO and get their results from there.
That is all you need to know.
Source: I am in the Top 10 AI SEO Experts listed by ChatGPT – usually first.
LLMs.txt files don’t control crawling. They are nothing like a robots.txt file.
And no, you don’t need to specifically allow them in robots.txt.
Automod has automatically removed this content. Your comment karma from this subreddit is low. Please engage with other threads before posting or improve your Contributor Quality Score on Reddit (CQS). To improve your CQS, focus on commenting over posting and avoid low-quality, reproduced posts across multiple subreddits.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/SEO) if you have any questions or concerns.*
Using “user-agent: *” avoids you needing to list bots individually btw… without that star your first one might not even be doing anything….
Edit: since you didn’t specify a user agent