scrapy 报错 ERROR: Error processing

报这种错误的时候要顺着后边的提示信息找到错误文件位置

一般来说是最后一条

例如报错信息是：

2017-04-05 19:38:42 [scrapy.core.scraper] ERROR: Error processing {'pic': [u'http://img.ivsky.com/img/tupian/li/201612/06/jianggushi_de_munv.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/06/jiaobu_yangsheng_spa-006.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/jianshen_de_nannv-002.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/jinfa_meinv-017.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/06/haidi_qianshui_renwu-013.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/06/kuaile_de_ertong-005.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/changqun_meinv_mote-002.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/yunfu_he_ertong-001.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/anjing_de_diaoyu.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/boshisheng_biye-011.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/yisheng_he_bingren-008.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/qi_zixingche_deren-002.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/yunfu-007.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/meinv_gongzuo-006.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/zhiye_bangong_nvxing.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/04/quanjia_jujia_wanshua-013.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/05/kuaile_ertong_yeying-005.jpg',
         u'http://img.ivsky.com/img/tupian/li/201612/04/shishang_qingnian-004.jpg']}
Traceback (most recent call last):
  File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 653, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "F:\code\spider\scrapy_spider\small_spider\small_spider\pipelines.py", line 14, in process_item
    urllib.request.urlretrieve(i,'z_p-'+pic_name)
AttributeError: 'module' object has no attribute 'request'

这里直接定位到最后几行：

File "F:\code\spider\scrapy_spider\small_spider\small_spider\pipelines.py", line 14, in process_item
    urllib.request.urlretrieve(i,'z_p-'+pic_name)
AttributeError: 'module' object has no attribute 'request'

可以看到是 pipelines.py 这个文件的 14 行出了错误

定位到相应代码位置：

# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: http://doc.scrapy.org/en/latest/topics/item-pipeline.html
import urllib

class SmallSpiderPipeline(object):
    def process_item(self, item, spider):
        for i in item['pic']:
            print(i)
            pic_name = i.split('/')[-1]
            urllib.request.urlretrieve(i,'z_p-'+pic_name)
        # return item

可以看到 14 行中的 urllib.request 错误，应改为urllib

修改相应错误后，再次执行完美通过