以前寫抓取網頁的代碼喜歡用ReadToEnd,因為簡單省事,後來發現,在爬取網頁的時候,如果網速很慢,ReadToEnd超時的幾率很大。使用Read改寫後,超時幾率大大減小,完整代碼如下: /// <summary> /// HttpPost /// </summary> public static ...
以前寫抓取網頁的代碼喜歡用ReadToEnd,因為簡單省事,後來發現,在爬取網頁的時候,如果網速很慢,ReadToEnd超時的幾率很大。使用Read改寫後,超時幾率大大減小,完整代碼如下:
/// <summary> /// HttpPost /// </summary> public static string HttpPost(string url, string data) { byte[] bArr = ASCIIEncoding.UTF8.GetBytes(data); // 設置參數 HttpWebRequest request = WebRequest.Create(url) as HttpWebRequest; request.CookieContainer = m_Cookie; request.Method = "POST"; request.ContentType = "application/x-www-form-urlencoded"; request.ContentLength = bArr.Length; request.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0)"; Stream postStream = request.GetRequestStream(); postStream.Write(bArr, 0, bArr.Length); postStream.Close(); //發送請求並獲取相應回應數據 HttpWebResponse response = request.GetResponse() as HttpWebResponse; //直到request.GetResponse()程式才開始向目標網頁發送Post請求 Stream responseStream = response.GetResponseStream(); //返回結果網頁(html)代碼 MemoryStream memoryStream = new MemoryStream(); bArr = new byte[1024]; int size = responseStream.Read(bArr, 0, (int)bArr.Length); while (size > 0) { memoryStream.Write(bArr, 0, size); size = responseStream.Read(bArr, 0, (int)bArr.Length); Thread.Sleep(1); } string content = Encoding.UTF8.GetString(memoryStream.ToArray()); return content; }View Code
代碼中Thread.Sleep(1);也可以去掉。