You're looking to add some additional bots but don't want to look for all the information? You've come to the right place.
Here is a list (which we will continue to add to) of bots you may be interested in adding to your Robots-Only deployment. This link provides instructions for enabling Robots-Only deployment, and customizing the list of robots.
General Bots
Bot Name (User Agent Token) | Full User Agent String |
360Spider | Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider |
AhrefsBot | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) |
AOLBuild | Mozilla/5.0 (compatible; MSIE 9.0; AOL 9.7; AOLBuild 4343.19; Windows NT 6.1; WOW64; Trident/5.0; FunWebProducts) |
Ask Jeeves/Teoma | Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml) |
botify | Mozilla/5.0 (compatible; botify; http://botify.com) |
ContentKing | Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com) |
DuckDuckBot | Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot) |
Baiduspider | Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) |
Qwantify | Mozilla/5.0 (compatible; Qwantify/Bleriot/1.2.1; +https://help.qwant.com/bot) |
SemrushBot | Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html) |
Daum | Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966) |
SeznamBot | Mozilla/5.0 (compatible; SeznamBot/4.0; +http://napoveda.seznam.cz/seznambot-intro/) |
Screaming Frog SEO Spider | Screaming Frog SEO Spider/5.1 |
LinkedInBot | LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com) |
Sogou web spider | Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) |
Twitterbot | Twitterbot/1.0 |
ichiro | ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html) |
Yeti | Mozilla/5.0 (compatible; Yeti/1.1; +https://naver.me/spd) |
AI Bots
There are also a number of AI search engines and chat bots using their own crawlers to collect data for training purposes.
Bot Name (User Agent Token) | Full User Agent String |
ChatGPT-User | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot) |
Diffbot | Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com) |
GPTBot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot) |
OAI-SearchBot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot) |
PerplexityBot | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) |
Note that while there are sometimes multiple different full user agent strings for a given Bot, we only need one to be added for the custom bots' deployment to work.
If you have additional suggestions for this list, please don't hesitate to reach out to support@schemaapp.com.
Was this article helpful?
That’s Great!
Thank you for your feedback
Sorry! We couldn't be helpful
Thank you for your feedback
Feedback sent
We appreciate your effort and will try to fix the article