鍍金池/ 問答/人工智能  Python/ scrapy 執(zhí)行了多個sprider

scrapy 執(zhí)行了多個sprider

scrapy 執(zhí)行了多個sprider
圖片描述

結(jié)構(gòu)如上圖
run函數(shù)代碼如下:

from scrapy import cmdline
cmdline.execute('scrapy crawl testform'.split())

tesform代碼如下:

import scrapy
import requests
import json
class testform(scrapy.Spider):
    name = 'testform'
    def start_requests(self):
        url='https://www.jiqizhixin.com/graphql'
        headers={
            'Accept': '*/*',
            'Accept-Encoding': 'gzip, deflate, br',
            'Accept-Language': 'zh-CN,zh;q=0.9',
            'Connection': 'keep-alive',
            'Content-Length': '320',
            'Content-Type': "application/x-www-form-urlencoded",
            'Cookie': '_Synced_session=YkdBYVlmcTYyc08zKzROUENpN081czhWblN2TGdBZEVWUGFCQWhHMi9VampHTTBXTEVYbzFOWS8reGxaQ0kxa0x5V1AzaFVPV09ob1FMaTNpdWN1WTg1VjUzN2EyWjRuK09CTGR6NXg1WURMMWtlQTMyZGpNcGtjZkZGOXRrZXdNUFJEVVp5RTVZcFJLNmpjQnAzUC9FRFNnSVJySlN2bDRLUUhxaUxXZ05WdTZPNHZXNVNYcmFIZTdSa21sKzNILS1uajBONjVGaWtvdFcrK3VwRU1oSlZnPT0%3D--f0f9f2de55f2a09c8c29ff407cb0ddcf758c546b; _gat=1; _ga=GA1.2.416194123.1514536384; _gid=GA1.2.218109006.1514536384',
            # 'Content-Type': "application/json",
            'Host': 'www.jiqizhixin.com',
            'Origin': 'https://www.jiqizhixin.com',
            'Referer': 'https://www.jiqizhixin.com/categories/zi-xun',
            'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36',
            'X-CSRF-Token': 'DfeIz+CdfLWv86lTgKmBu5xDl+hpbIxoE8wmxWIuhzECfwvWfO62+uO2dtqXaKh4nzSM4xRh3/TsTKrEbw73/Q=='
        }
        payload={
            'operationName': 'Search',
            'query': "query Search($cursor: String, $count: Int, $keywords: String!, $filter_tags: [String]) {\n  elastic_search(first: $count, after: $cursor, keywords: $keywords, filter_tags: $filter_tags) {\n    edges {\n      node {\n        id\n        title\n        tag_list\n        content\n        author_name\n        categories\n        cover_image_url\n        published_at\n        friendly_path\n        __typename\n      }\n      __typename\n    }\n    total_count\n    pageInfo {\n      endCursor\n      hasNextPage\n      __typename\n    }\n    __typename\n  }\n}\n",
            'variables': {'count': 50, 'keywords':'NLP', 'filter_tags': []},
        }
        req=requests.post(url,json=payload,headers=headers)
        # req = requests.post(url, data=json.dumps(payload), headers=headers)
        yield scrapy.FormRequest(url,formdata=payload,headers=headers)
    def parse(self, response):
        print(response)

testlogin里面代碼是print(111)一句話
輸出結(jié)果如下:
圖片描述

可見先執(zhí)行了testlogin再執(zhí)行testform,為什么呢?????????????????

回答
編輯回答
厭遇

這個與scrapy.spiderloader.SpiderLoader類有關(guān),這個類在實例化時會import所有的spider

2017年5月28日 03:57