鍍金池/ 問答/Java  Linux  HTML/ httpClient請求得到的頁面和Chrome監(jiān)控到的返回頁面相比,少了一些內(nèi)

httpClient請求得到的頁面和Chrome監(jiān)控到的返回頁面相比,少了一些內(nèi)容?

背景

用httpClient爬取頁面的時候發(fā)現(xiàn)爬到的頁面和chrome監(jiān)控到的少了幾行。

Chrome監(jiān)控到的:

clipboard.png

我抓到的:

clipboard.png

可以看到下面比上面少了一些內(nèi)容。

問題

  1. 我第一感覺是js生成的,請問chrome監(jiān)控到的頁面是經(jīng)過js腳本加工之后的嗎,不應該請求返回的原始頁面嗎?
  2. 我在source中看了所有的js腳本,都沒有查找到相關的添加元素的腳本代碼。請問這個問題的可能原因是什么,我這樣找對嗎?

clipboard.png

補充1

以下是我模擬參數(shù)的代碼

/以下參數(shù)除去了抓包時看到的undefined的值
                nvps.add(new BasicNameValuePair("businessType", "PCIS"));
                nvps.add(new BasicNameValuePair("businessNo", ePosPaymentReturnVO.getBusinessNo()));
                nvps.add(new BasicNameValuePair("customerName", ePosPaymentReturnVO.getCustomerName()));
                nvps.add(new BasicNameValuePair("currencyNo", ePosPaymentReturnVO.getCurrencyNo()));
                nvps.add(new BasicNameValuePair("amount", ePosPaymentReturnVO.getAmount()));
                nvps.add(new BasicNameValuePair("regionCode", ePosPaymentReturnVO.getRegionCode()));
                nvps.add(new BasicNameValuePair("insuredName", ePosPaymentReturnVO.getInsuredName()));
                nvps.add(new BasicNameValuePair("tellerNo", ePosPaymentReturnVO.getTellerNo()));
                nvps.add(new BasicNameValuePair("branchNo", ePosPaymentReturnVO.getBranchNo()));
                nvps.add(new BasicNameValuePair("documentNo", ""));
                nvps.add(new BasicNameValuePair("callBackInfo", ""));
                nvps.add(new BasicNameValuePair("certPubKey", ePosPaymentReturnVO.getCertPubKey()));
                nvps.add(new BasicNameValuePair("signData", ePosPaymentReturnVO.getSignData()));
                nvps.add(new BasicNameValuePair("applicantName", ePosPaymentReturnVO.getApplicantName()));
                nvps.add(new BasicNameValuePair("applicantCertificateType", ePosPaymentReturnVO.getApplicantCertificateType()));
                nvps.add(new BasicNameValuePair("applicantCertificateNo", ePosPaymentReturnVO.getApplicantCertificateNo()));
                nvps.add(new BasicNameValuePair("ncpInsuredName", ePosPaymentReturnVO.getNcpInsuredName()));
                nvps.add(new BasicNameValuePair("insuredCertificateType", ePosPaymentReturnVO.getInsuredCertificateType()));
                nvps.add(new BasicNameValuePair("insuredCertificateNo", ePosPaymentReturnVO.getInsuredCertificateNo()));
                nvps.add(new BasicNameValuePair("networkFlag", ePosPaymentReturnVO.getNetworkFlag()));
                nvps.add(new BasicNameValuePair("applicantTelephone", ePosPaymentReturnVO.getApplicantTelephone()));
                nvps.add(new BasicNameValuePair("channelSource", ePosPaymentReturnVO.getChannelSource()));
                nvps.add(new BasicNameValuePair("businessTranChnl", ePosPaymentReturnVO.getBusinessTranChnl()));
                nvps.add(new BasicNameValuePair("businessTranCode", ePosPaymentReturnVO.getBusinessTranCode()));
                nvps.add(new BasicNameValuePair("prePayAmount", ""));
                nvps.add(new BasicNameValuePair("isSocialSecurityPay", ePosPaymentReturnVO.getIsSocialSecurityPay()));
                nvps.add(new BasicNameValuePair("customerCertType", ePosPaymentReturnVO.getCustomerCertType()));
                nvps.add(new BasicNameValuePair("customerCertId", ePosPaymentReturnVO.getCustomerCertId()));
                nvps.add(new BasicNameValuePair("userName", ePosPaymentReturnVO.getUserName()));

以下是在chrome中看到的請求參數(shù)

businessType:PCIS
businessNo:12010088000293602413
customerName:(unable to decode value)
currencyNo:RMB
amount:10429
regionCode:000000
insuredName:(unable to decode value)
tellerNo:YWSHP-00001
branchNo:2120971
documentNo:
callBackInfo:
certPubKey:LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVqRENDQTNTZ0F3SUJBZ0lNTjRBVzdPVDM0
dXVkZDFHRk1BMEdDU3FHU0liM0RRRUJCUVVBTUNzeEN6QUoKQmdOVkJBWVRBa05PTVF3d0NnWURW
UVFLREFOVFJFSXhEakFNQmdOVkJBTU1CVk5FUWtOQk1CNFhEVEUwTURVeApNekF5TlRjd09Gb1hE
VEU1TURVeE1qQXlOVGN3T0Zvd2J6RUxNQWtHQTFVRUJoTUNRMDR4Q3pBSkJnTlZCQWdUCkFsTmFN
UTR3REFZRFZRUUhEQVZNVDBOQlRERU5NQXNHQTFVRUNnd0VVRUZKUXpFVE1CRUdBMVVFQ3d3S1JV
VmYKVTBsSFRrbE9SekVmTUIwR0ExVUVBd3dXWlhCamFYTXRZWFYwYnk1d1lXbGpMbU52YlM1amJq
Q0JuekFOQmdrcQpoa2lHOXcwQkFRRUZBQU9CalFBd2dZa0NnWUVBbWN6WmdnOUx4UTZLN1pvWVhS
b2ZXcEYxLzY3MGttekFWSlZQCjF3WXQzVVZVMXNXTDB0RGhZUFdmZHp3ZzNFbzNVL0toTldDNzFS
dm8xSXRsQWlqUWI1cDZ0Rk1vL0xXZmtxOU8KMHpBS00xM2l3MjNlZytkcjI4RHhLKzZIbXJBNkVa
dHkwKzFkVWhzWUZiMXN2S3dDdUFIWjJ5WWYvU2ZrcDBIVwpwSkZoa3lrQ0F3RUFBYU9DQWU0d2dn
SHFNQThHQTFVZEV3RUJBQVFGTUFNQkFRQXdEZ1lEVlIwUEFRRUFCQVFECkFnRENNQlFHQ1dDR1NB
R0crRUlCQVFFQkFBUUVBd0lBNGpBZ0JnTlZIU1VCQVFBRUZqQVVCZ2dyQmdFRkJRY0QKQVFZSUt3
WUJCUVVIQXdJd0lnWURWUjBqQVFFQUJCZ3dGb0FVN0tZVEM2QkIyTU9DZEtpN2RrTW9IenBjRkJj
dwpnWjhHQ0NzR0FRVUZCd0VCQVFFQUJJR1BNSUdNTUlHSkJnZ3JCZ0VGQlFjd0FvWjliR1JoY0Rv
dkx6RXdMakl1Ck1UQXhMakUxT2pRNE9TOURUajFUUkVKRFFTeERUajFUUkVKRFFTeFBWVDFqUVVO
bGNuUnBabWxqWVhSbGN5eHYKUFhOa1ltTmhMR005WTI0L1kwRkRaWEowYVdacFkyRjBaVDlpWVhO
bFAyOWlhbVZqZEVOc1lYTnpQV05sY25ScApabWxqWVhScGIyNUJkWFJvYjNKcGRIa3dnYVlHQTFV
ZEh3RUJBQVNCbXpDQm1EQ0JsYUNCa3FDQmo0YUJqR3hrCllYQTZMeTh4TUM0eUxqRXdNUzR4TlRv
ME9Ea3ZRMDQ5VTBSQ1EwRXNRMDQ5VTBSQ1EwRXNiM1U5UTFKTVJHbHoKZEhKcFluVjBaVkJ2YVc1
MGN5eHZQWE5rWW1OaExHTTlZMjQvWTJWeWRHbG1hV05oZEdWU1pYWnZZMkYwYVc5dQpUR2x6ZEQ5
aVlYTmxQMjlpYW1WamRHTnNZWE56UFdOU1RFUnBjM1J5YVdKMWRHbHZibEJ2YVc1ME1DQUdBMVVk
CkRnRUJBQVFXQkJTODBieXE5ejF0eEx2cWRLYmJldElOWnkxU2hUQU5CZ2txaGtpRzl3MEJBUVVG
QUFPQ0FRRUEKYjRBSVljRnExRXF0OHhJZ1pleDczdGx6aGFmNEgyTXlyS0Z0dG1ubFRFc3RZbkpi
UEdJQ0x6MitNcU1FV0JYcAp4amw3M0JNYUROR0xvMkIremZrVTJxVlk3eEh4b1dNcUR1Zm9EUVRR
Sm5SU2xHRGFKM3QyMDV2dmZ2NWEramFnCk9KeW9kZ0hCUXhrbEJ5alNqSityaFpRdEV1UGNJVlhI
U2t6MVYzK0hFL3piWE9QUVdkUVVUQWZjU25kMlpIOFYKbmxrTHFBTU0rbWI3ZlJTUDJCelFkVmIv
ZFVqSlZpckhuMGN5SzZ3MUwzVTY0dEhXWm5jdE9NKzdWUGx0V09jaQpwcFF2M21mOEwzUmxmRnVC
dXE3aUdBQlBlMFZyT2IwVXdEczFGSGNRSHN1L21kdW5kenU5YURESnc2UHVpQTVuCnlKbWU2R29D
eWQ2a0tTMzFxRFova3c9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
signData:J5sgq0DZJj4GK6cVdQe3B0Pt+KuM1mLMUSapFxHq5XrpHbsv1KHWfGikQn9mjez7W9p12cq9QOTS
+5UNkzO/dVoSv92GwN8i115vwIMkpA3Zp5PksOn66obtCOLD7CdZAeB57QtOYvlFqEI4RDZ8oqLk
eYROa0P4f5AKxa1BRu8=
callBackURL:undefined
applicantName:(unable to decode value)
applicantCertificateType:01
applicantCertificateNo:330183199009264314
ncpInsuredName:(unable to decode value)
insuredCertificateType:01
insuredCertificateNo:330183199009264314
prepaidAccountId:undefined
dataSource:33
payType:undefined
prepaidAccountType:undefined
networkFlag:02
applicantTelephone:15057179897
channelSource:G
businessTranChnl:EPO
businessTranCode:000052
prePayAmount:
isContainTax:undefined
vehicleNum:undefined
bargainNum:undefined
departmentName:undefined
operationByName:undefined
inputByName:undefined
undrContractName:undefined
payNo:undefined
payMode:undefined
paBankAccount:undefined
paBankAccountName:undefined
vehicleTaxCountStr:undefined
collectAmountStr:undefined
paymentEndDate:undefined
circPaymentNo:undefined
printFlag:undefined
departmentCode:undefined
isSocialSecurityPay:N
socialSecurityPayInfo:undefined
customerCertType:01
customerCertId:330183199009264314

可以看undefined的參數(shù)是沒有加的。這樣有問題嗎?

請各位大佬相助。

先謝過了。

回答
編輯回答
離觴

ctrl+u 看一下他的源代碼,就是不包含 js 生成的源代碼,

clipboard.png

這個就是你抓取所獲取的內(nèi)容,應該是不包含這部分的:

clipboard.png

chrome監(jiān)控到的頁面是經(jīng)過js腳本加工之后的嗎?

當然,沒有經(jīng)過 js 修飾過得,多難用;

我在source中看了所有的js腳本,都沒有查找到相關的添加元素的腳本代碼。請問這個問題的可能原因是什么,我這樣找對嗎?

如果你前面 ctrl+u 確實證明我說的是對的話,那這段多出來的代碼就是 js 生成的,你沒有找到很正常呀,那么多代碼你一行行看了,都明白什么意思?不太可能吧

2017年6月28日 04:37
編輯回答
涼心人

一直忘了,現(xiàn)在自答一下,之所以請求頁面不完整。是因為請求參數(shù)沒有模擬成功,當前post請求的參數(shù),是由上一步的請求得到的數(shù)據(jù)進行模擬的,其中有一個netFlag的參數(shù),在上一步請求的返回參數(shù)中值是"Internet",但是在實際頁面請求時,在頁面上被轉(zhuǎn)成了對應的code值"02",我在模擬這個值得時候直接就是拿了上一步返回參數(shù)里的值,也就是"internet",而實際上應該是要返回"02"。修改了之后就請求成功了。

2017年10月25日 02:00
編輯回答
慢半拍

對于httpclient而言,是在io層面,數(shù)據(jù)都只是文件,字符。而瀏覽器的作用就是解釋HTML,css,js代碼,他會執(zhí)行頁面嵌入的腳本,渲染頁面等等。

2017年12月2日 03:53