pymysql讀取大數(shù)據(jù)時(shí)候，用生成器生成數(shù)據(jù)異常退出

前提：本人要處理一個(gè)mysql表，數(shù)據(jù)大概500-600萬的樣子，通過遍歷表進(jìn)行數(shù)據(jù)的加工和清洗
操作:使用python3導(dǎo)入pymysql包進(jìn)行操作，遇到二個(gè)問題，第一數(shù)據(jù)量太大，如果用常規(guī)連接然后不管是fetchone,fetchmany,fetchall都將占用大量加載時(shí)間,第二全部加載到內(nèi)存，內(nèi)存占用太高，（fetchone和fetchmany原理差不多,查了資料都說是先返回全部數(shù)據(jù)，然后遍歷）,不管fetchone和fetchmany說的對(duì)不對(duì)，占用大量時(shí)間，消耗大量內(nèi)存是事實(shí)情況
解決：通過查看pymysql官方文檔，官方文檔寫的是用無緩沖游標(biāo)pymysql.cursors.SSCursor，這樣速度確實(shí)加快了，我測(cè)試了代碼確實(shí)加載速度很快
遇到的問題：當(dāng)我用生成器生成讀取的數(shù)據(jù)然后進(jìn)行進(jìn)一步使用時(shí)候，數(shù)據(jù)沒有讀完就報(bào)錯(cuò)誤退出，而且我只是用300條測(cè)試數(shù)據(jù)就出錯(cuò),報(bào)錯(cuò)信息提示是游標(biāo)讀取數(shù)據(jù)為None，然后異常退出，關(guān)鍵是每次讀取數(shù)據(jù)量不等，然后報(bào)錯(cuò)，當(dāng)我取消無緩沖游標(biāo)方法，數(shù)據(jù)正確讀完無異常

import pymysql
import time

class Check():
    def __init__(self, host, user, password,db,port=3306,charset="utf8"):
        self.host = host
        self.user = user
        self.password = password
        self.database = db
        self.port = port
        self.charset = charset
        try:
            self.conn = pymysql.connect(
                host=self.host,
                db=self.database,
                user=self.user,
                passwd=self.password,
                port=self.port,
                charset=self.charset,
                cursorclass=pymysql.cursors.SSCursor)  #無緩沖游標(biāo)，當(dāng)取消時(shí)候無異常
            self.cursor = self.conn.cursor()
        except:
            print('Mysql connect error!')

    def setDB(self,dbname):

        try:
            self.conn.select_db(dbname)
        except:
            print('select DB error')

    def queryAll(self, sql):

        try:
            self.cursor.execute(sql)
            while True:
                rows = self.cursor.fetchone()
                if rows:
                    yield rows
                    time.sleep(5)   #模擬其他操作
                else:
                    break
        except:
            print ('Get Data Error!')

    def commit(self):
        self.conn.commit()

    def close(self):
        self.cursor.close()
        self.conn.close()


if __name__ == '__main__':
    test = Check('127.0.0.1','root','123456','test')
    posts = test.queryAll('select * from posts ')
    a = 0
    for post in posts:
        print (post)
        a +=1
        print (a)
    print ("all down,%s",a)