如题,按照下面两种方式设置,搜索引擎的爬虫还是会被屏蔽掉

  1. valid_referers *.aaa.com server_names ~\.google\. ~\.baidu\.;
  1. valid_referers .aaaa.com .google.com *.baidu.com;

123.125.71.105 - - [17/Jun/2017:00:45:51 +0800] "GET /******** HTTP/1.1" 403 162 "-" "Baiduspider-image+(+http://www.baidu.com/search/spider.htm)" -