主流搜索引擎 spider 蜘蛛 User-Agent 标识集
warning:
这篇文章距离上次修改已过224天，其中的内容可能已经有所变动。
| Adsbot-Google | Adsbot-Google | |
| AlexaBot | Alexa | |
| AOLBot | sqworm | |
| BaiduBot | Baiduspider | |
| GoogleBot | googlebot | http://www.googlebot.com/bot.html |
| Googlebot-Image | Googlebot-Image | |
| Googlebot-Mobile | Googlebot-Mobile | |
| IasksBot | iaskspider | |
| LycosBot | lycos | |
| Mediapartners-Google | Mediapartners-Google | |
| MSN Bot | msnbot | http://search.msn.com/msnbot.htm |
| RobozillaBot | robozilla | |
| SohuBot | sohu-search | |
| SosoBot | Sosospider | |
| SougouBot | Sogou web spider | http://www.sogou.com/docs/help/webmasters.htm#07 |
| Yahoo Slurp | slurp | http://help.yahoo.com/l/us/yahoo/search/webcrawler/index.html |
| Yahoo-AdCrawler | AdCrawler | |
| YoudaoBot | YodaoBot | http://www.yodao.com/help/webmaster/spider |