What Custom Bots should I add to my Robots-only deployment?

Modified on Fri, 26 Jul at 2:33 PM

You're looking to add some additional bots but don't want to look for all the information? You've come to the right place. 


Here is a list (which we will continue to add to) of bots you may be interested in adding to your Robots-Only deployment. This link provides instructions for enabling Robots-Only deployment, and customizing the list of robots. 


General Bots

Bot Name (User Agent Token)Full User Agent String
360SpiderMozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider
AhrefsBotMozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
AOLBuildMozilla/5.0 (compatible; MSIE 9.0; AOL 9.7; AOLBuild 4343.19; Windows NT 6.1; WOW64; Trident/5.0; FunWebProducts) 
Ask Jeeves/TeomaMozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)
botifyMozilla/5.0 (compatible; botify; http://botify.com)
ContentKingMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)

DuckDuckBot

Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)
BaiduspiderMozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
QwantifyMozilla/5.0 (compatible; Qwantify/Bleriot/1.2.1; +https://help.qwant.com/bot)
SemrushBotMozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
DaumMozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)
SeznamBotMozilla/5.0 (compatible; SeznamBot/4.0; +http://napoveda.seznam.cz/seznambot-intro/)
Screaming Frog SEO SpiderScreaming Frog SEO Spider/5.1
LinkedInBotLinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)
Sogou web spiderSogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
TwitterbotTwitterbot/1.0
ichiroichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)
YetiMozilla/5.0 (compatible; Yeti/1.1; +https://naver.me/spd)


AI Bots

There are also a number of AI search engines and chat bots using their own crawlers to collect data for training purposes. 


Bot Name (User Agent Token)Full User Agent String
ChatGPT-User
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot)
DiffbotMozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
GPTBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)
OAI-SearchBot    
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)
PerplexityBotMozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)




Note that while there are sometimes multiple different full user agent strings for a given Bot, we only need one to be added for the custom bots' deployment to work.


If you have additional suggestions for this list, please don't hesitate to reach out to support@schemaapp.com


Was this article helpful?

That’s Great!

Thank you for your feedback

Sorry! We couldn't be helpful

Thank you for your feedback

Let us know how can we improve this article!

Select at least one of the reasons
CAPTCHA verification is required.

Feedback sent

We appreciate your effort and will try to fix the article