使用Spark分析拉勾網招聘信息(二): 獲取數據

来源:http://www.cnblogs.com/ios122/archive/2016/09/01/5828673.html
-Advertisement-
Play Games

要獲取什麼樣的數據? 我們要獲取的數據,是指那些公開的,可以輕易地獲取地數據.如果你有完整的數據集,肯定是極好的,但一般都很難通過還算正當的方式輕易獲取.單就本系列文章要研究的實時招聘信息來講,能獲取最近一個月的相關信息,已是足矣. 如何獲取數據? 爬蟲,也是可以的,作為一個備選方案.但是,我註意到 ...


要獲取什麼樣的數據?

我們要獲取的數據,是指那些公開的,可以輕易地獲取地數據.如果你有完整的數據集,肯定是極好的,但一般都很難通過還算正當的方式輕易獲取.單就本系列文章要研究的實時招聘信息來講,能獲取最近一個月的相關信息,已是足矣.

如何獲取數據?

爬蟲,也是可以的,作為一個備選方案.但是,我註意到拉勾網本身的數據,是通過ajax請求更新的,所以批量獲取變得更加簡單.基於ajax請求來獲取數據,方式有很多,這裡我演示其中的自認為較為簡單通用的一種: 使用 curl 模擬 ajax 請求獲取數據.

註意,以下的步驟演示全部基於 Mac 版的 ** Google Chrome** 瀏覽器,其他瀏覽器部分操作可能會有些許差異.最後一步會給出 提取出的通用 curl 腳本,直接其實也是可以的,如果對步驟不是很關心.

1.找到目標城市和目標職位,然後按"最新排序",參考鏈接: http://www.lagou.com/jobs/list_iOS?px=new&city=北京#order

0-0.png

2.雙指擊/右擊 頁面,彈出快捷菜單,選擇"檢查",以進入瀏覽器調試界面,切換到調試器的 network -> xhr 標簽下.

0_1.png

3.cmd + R 刷新頁面,此時會捕捉到此頁面發出的xhr請求.找到 http://www.lagou.com/jobs/positionAjax.json 開頭的請求,並雙指擊/右擊,選擇 copy as cUrl.

這個 curl代碼非常長,對於本次分析來說,最關鍵的是 末尾的 pn=1&kd=iOS,分別代表著頁面和職位,動態設置,即可獲取更多職位的更多數據了,文章的其他篇幅,會單獨分析.

curl 'http://www.lagou.com/jobs/positionAjax.json?px=new&city=%E5%8C%97%E4%BA%AC&needAddtionalResult=false' -H 'Cookie: user_trace_token=20160522122749-8a4d6717-1fd5-11e6-963e-5254005c3644; LGUID=20160522122749-8a4d6cb3-1fd5-11e6-963e-5254005c3644; tencentSig=8513357824; LGMOID=20160818212815-33C56329AA2FB6D809D557FD6CC1DE3C; JSESSIONID=E0C5692414B160F3BEFFCFF8D240B693; _gat=1; PRE_UTM=; PRE_HOST=; PRE_SITE=; PRE_LAND=http%3A%2F%2Fwww.lagou.com%2Fjobs%2Flist_iOS%3Fpx%3Dnew%26city%3D%25E5%258C%2597%25E4%25BA%25AC; ctk=1472587366; Hm_lvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1470032014,1470032429,1471526897,1471752875; Hm_lpvt_4233e74dff0ae5bd0a3d81c6ccf756e6=1472587368; LGSID=20160831035434-92fffed3-6eeb-11e6-8c02-5254005c3644; LGRID=20160831040247-b90762b6-6eec-11e6-a745-525400f775ce; _ga=GA1.2.1654849521.1463891270; SEARCH_ID=506c8097626343ab9b5a2c1139807a2c' -H 'Origin: http://www.lagou.com' -H 'X-Anit-Forge-Code: 0' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: zh-CN,zh;q=0.8,en;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Cache-Control: max-age=0' -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' -H 'X-Anit-Forge-Token: None' -H 'Referer: http://www.lagou.com/jobs/list_iOS?px=new&city=%E5%8C%97%E4%BA%AC' --data 'first=true&pn=1&kd=iOS' --compressed

0_2.png

4.講上一步中的curl指令複製到終端,橋下回車鍵,即可看到輸出.

{"success":true,"requestId":null,"msg":null,"resubmitToken":null,"content":{"pageNo":1,"pageSize":15,"positionResult":{"totalCount":974,"resultSize":15,"locationInfo":{"city":"北京","district":null,"queryByGisCode":false,"businessZone":null,"locationCode":null},"queryAnalysisInfo":{"positionName":"ios","companyName":null,"usefulCompany":false,"industryName":null},"strategyProperty":{"name":"dm-csearch-newSimScorer","id":1},"result":[{"companyId":129801,"companyShortName":"言之有物科技","createTime":"2016-08-30 19:28:12","positionId":1857486,"positionAdvantage":"一線公司，技術驅動，免費三餐，超期望回報","salary":"25k-50k","score":0,"workYear":"不限","education":"本科","city":"北京","positionName":"iOS高級研發工程師/Lead","companyLogo":"i/image/M00/43/4E/CgqKkVeDGsuAXz0gAAA4XeGAAHQ390.png","financeStage":"成長型(A輪)","industryField":"移動互聯網,電子商務","jobNature":"全職","approve":1,"companySize":"15-50人","district":null,"companyLabelList":["股票期權","扁平管理","美女多","領導好"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"19:28發佈","gradeDescription":null,"companyFullName":"北京言之有物科技有限公司","businessZones":null,"imState":"today","lastLogin":1472556472000,"publisherId":5092848,"explain":null,"plus":null,"pcShow":0},{"companyId":133,"companyShortName":"獵豹移動","createTime":"2016-08-30 19:09:34","positionId":2151896,"positionAdvantage":"明星產品 超贊年終獎 靠譜領導","salary":"15k-30k","score":0,"workYear":"1-3年","education":"本科","city":"北京","positionName":"iOS","companyLogo":"image1/M00/39/70/CgYXBlWo3nqABJTsAADJ3hn5gmE062.jpg","financeStage":"上市公司","industryField":"移動互聯網,信息安全","jobNature":"全職","approve":1,"companySize":"500-2000人","district":"朝陽區","companyLabelList":["帶薪年假","美女前臺","超贊年終獎","一公裡工作圈"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"19:09發佈","gradeDescription":null,"companyFullName":"北京金山網路科技有限公司","businessZones":["姚家園","十里堡","高碑店"],"imState":"today","lastLogin":1472555392000,"publisherId":129969,"explain":null,"plus":null,"pcShow":0},{"companyId":107608,"companyShortName":"MUM電腦","createTime":"2016-08-30 19:03:24","positionId":1963945,"positionAdvantage":"幫助程式員赴美做IT，享受高薪高品質生活","salary":"10k-20k","score":0,"workYear":"不限","education":"本科","city":"北京","positionName":"IOS程式員赴美項目推廣員","companyLogo":"i/image/M00/00/C2/CgqKkVZVHmSAWPtRAASUg0iUVuI932.jpg","financeStage":"初創型(不需要融資)","industryField":"教育","jobNature":"全職","approve":0,"companySize":"少於15人","district":"昌平區","companyLabelList":["赴美工作","美元薪水","告別996","技術前沿"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"19:03發佈","gradeDescription":null,"companyFullName":"北京瑪赫西電腦教育咨詢有限公司","businessZones":null,"imState":"disabled","lastLogin":1472558059000,"publisherId":5179699,"explain":null,"plus":null,"pcShow":0},{"companyId":67576,"companyShortName":"車滿滿","createTime":"2016-08-30 18:47:30","positionId":2307877,"positionAdvantage":"期權","salary":"20k-25k","score":0,"workYear":"3-5年","education":"本科","city":"北京","positionName":"iOS高級開發工程師","companyLogo":"i/image/M00/01/47/Cgp3O1ZmYACABBpPAAGzVR5S-Ps906.png","financeStage":"成長型(A輪)","industryField":"移動互聯網","jobNature":"全職","approve":1,"companySize":"50-150人","district":"朝陽區","companyLabelList":["股票期權","技能培訓","彈性工作","定期體檢"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:47發佈","gradeDescription":null,"companyFullName":"車滿滿（北京）信息技術有限公司","businessZones":["建外大街","CBD","國貿"],"imState":"today","lastLogin":1472566873000,"publisherId":2116322,"explain":null,"plus":null,"pcShow":0},{"companyId":1575,"companyShortName":"百度","createTime":"2016-08-30 18:30:05","positionId":2307765,"positionAdvantage":"BAT 薪酬福利好","salary":"15k-25k","score":0,"workYear":"3-5年","education":"本科","city":"北京","positionName":"iOS移動開發","companyLogo":"image1/M00/00/06/CgYXBlTUWAWAOBXrAABGHHFb0q8748.jpg","financeStage":"上市公司","industryField":"移動互聯網,數據服務","jobNature":"全職","approve":1,"companySize":"2000人以上","district":null,"companyLabelList":["股票期權","彈性工作","五險一金","免費班車"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:30發佈","gradeDescription":null,"companyFullName":"百度線上網路技術（北京）有限公司","businessZones":null,"imState":"disabled","lastLogin":1472553001000,"publisherId":5705515,"explain":null,"plus":null,"pcShow":0},{"companyId":13321,"companyShortName":"FunPlus 趣加游戲","createTime":"2016-08-30 18:26:28","positionId":2240276,"positionAdvantage":"國際一線團隊，無限的成長空間，任你發揮","salary":"18k-36k","score":0,"workYear":"5-10年","education":"本科","city":"北京","positionName":"iOS 視頻處理工程師/高級工程師","companyLogo":"image1/M00/00/1A/Cgo8PFTUWFWAKE5aAABwJ1mgAYw423.png","financeStage":"成長型(B輪)","industryField":"游戲","jobNature":"全職","approve":0,"companySize":"150-500人","district":"海澱區","companyLabelList":["績效獎金","股票期權","專項獎金","五險一金"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:26發佈","gradeDescription":null,"companyFullName":"北京趣加科技有限公司","businessZones":["中關村","知春路","雙榆樹"],"imState":"today","lastLogin":1472552889000,"publisherId":285309,"explain":null,"plus":null,"pcShow":0},{"companyId":15111,"companyShortName":"聯拓天際","createTime":"2016-08-30 18:22:12","positionId":2307696,"positionAdvantage":"與其在別處仰望，不如在這裡並肩","salary":"15k-25k","score":0,"workYear":"3-5年","education":"本科","city":"北京","positionName":"iOS","companyLogo":"image1/M00/00/1D/Cgo8PFTUWGGAZQdjAADRNZVO9fc470.jpg","financeStage":"成熟型(不需要融資)","industryField":"電子商務","jobNature":"全職","approve":1,"companySize":"500-2000人","district":null,"companyLabelList":["五險一金","午餐補助","定期體檢","技能培訓"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:22發佈","gradeDescription":null,"companyFullName":"北京聯拓天際電子商務有限公司","businessZones":null,"imState":"today","lastLogin":1472552392000,"publisherId":1595082,"explain":null,"plus":null,"pcShow":0},{"companyId":119049,"companyShortName":"優久科技","createTime":"2016-08-30 18:15:29","positionId":1853231,"positionAdvantage":"良好的工作環境、成長平臺和工作伙伴","salary":"10k-18k","score":0,"workYear":"1-3年","education":"本科","city":"北京","positionName":"iOS","companyLogo":"i/image/M00/16/74/CgqKkVbvnVuAeC-YAAA_YSPyb5A166.jpg","financeStage":"初創型(天使輪)","industryField":"移動互聯網","jobNature":"全職","approve":0,"companySize":"少於15人","district":"海澱區","companyLabelList":["交通補助","通訊津貼","午餐補助"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:15發佈","gradeDescription":null,"companyFullName":"北京優久科技有限責任公司","businessZones":["中關村","知春路","人民大學"],"imState":"today","lastLogin":1472552013000,"publisherId":4427723,"explain":null,"plus":null,"pcShow":0},{"companyId":41878,"companyShortName":"商詢科技","createTime":"2016-08-30 18:14:06","positionId":2278393,"positionAdvantage":"微軟創業團隊，工程師文化！","salary":"10k-15k","score":0,"workYear":"1-3年","education":"本科","city":"北京","positionName":"iOS開發","companyLogo":"i/image/M00/24/22/Cgp3O1cZmpWAGslpAAA9MdgVNWU645.jpg","financeStage":"成長型(A輪)","industryField":"企業服務,數據服務","jobNature":"全職","approve":1,"companySize":"15-50人","district":"朝陽區","companyLabelList":["股票期權","人脈資源","辦公環境好","國際化團隊"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:14發佈","gradeDescription":null,"companyFullName":"北京商詢科技有限公司","businessZones":["姚家園"],"imState":"today","lastLogin":1472554153000,"publisherId":803257,"explain":null,"plus":null,"pcShow":0},{"companyId":5832,"companyShortName":"新浪微博","createTime":"2016-08-30 18:02:30","positionId":254885,"positionAdvantage":"億級別DAU，微博重點項目組","salary":"20k-40k","score":0,"workYear":"1-3年","education":"本科","city":"北京","positionName":"新浪微博iOS客戶端研發工程師","companyLogo":"image1/M00/00/0D/CgYXBlTUWCCAdkhOAABNgyvZQag818.jpg","financeStage":"上市公司","industryField":"移動互聯網","jobNature":"全職","approve":0,"companySize":"2000人以上","district":"海澱區","companyLabelList":["年底雙薪","專項獎金","股票期權","五險一金"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:02發佈","gradeDescription":null,"companyFullName":"微夢創科網路科技（中國）有限公司","businessZones":["西北旺","馬連窪","上地"],"imState":"disabled","lastLogin":1472556144000,"publisherId":561302,"explain":null,"plus":null,"pcShow":0},{"companyId":48321,"companyShortName":"合廣眾","createTime":"2016-08-30 18:00:40","positionId":2263615,"positionAdvantage":"老闆nice","salary":"10k-20k","score":0,"workYear":"3-5年","education":"本科","city":"北京","positionName":"iOS開發工程師","companyLogo":"i/image/M00/01/D6/CgqKkVZ496GAYypzAAAKATKLXuY379.png","financeStage":"初創型(天使輪)","industryField":"移動互聯網","jobNature":"全職","approve":0,"companySize":"50-150人","district":"海澱區","companyLabelList":["節日禮物","帶薪年假","績效獎金","崗位晉升"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"18:00發佈","gradeDescription":null,"companyFullName":"北京合廣眾文化發展有限公司","businessZones":["八里莊","定慧寺","四季青"],"imState":"today","lastLogin":1472550077000,"publisherId":3608518,"explain":null,"plus":null,"pcShow":0},{"companyId":38239,"companyShortName":"Keep","createTime":"2016-08-30 17:52:25","positionId":2076872,"positionAdvantage":"福利健全、北京工作居住證、C輪","salary":"25k-35k","score":0,"workYear":"5-10年","education":"本科","city":"北京","positionName":"iOS開發工程師","companyLogo":"image1/M00/0A/40/CgYXBlTun9KASqKdAAAs36QVurU409.png","financeStage":"成熟型(C輪)","industryField":"社交網路,文化娛樂","jobNature":"全職","approve":1,"companySize":"150-500人","district":null,"companyLabelList":["節日禮物","年度旅游","定期體檢","五險一金"],"adWord":0,"appShow":0,"deliver":0,"formatCreateTime":"17:52發佈","gradeDescription":null,"companyFullName":"北京卡路里科技有限公司","businessZones":null,"imState":"today","lastLogin":1472550738000,"publisherId":3425178,"explain":null,
您的分享是我們最大的動力!

-Advertisement-
Play Games
更多相關文章
  • 大數據環境下，關聯規則面臨的新挑戰。
    隨著互聯網、信息技術以及雲計算的高速發展，當今社會已進入了海量數據的時代。不管是移動通信、電商金融還是物聯網等各個領域，每天都會產生巨量的各種不同類型的數據。如何從這些龐大的、結構各異的、而又摻雜著大量雜訊的數據中提取出隱含在其中的具有一定意義的知識或規則，正是關聯規則提取的研究內容。 傳統的關聯規 ...
  • mysql中union與union all的區別
    當查詢表結構完全相同的多張表的數據時： 1.當查詢條件完全相同且不包括主鍵，此時用union查詢會過濾掉查詢出的重覆的記錄，及漏查記錄；使用union all進行查詢，則會查出所有的符合條件的記錄，保證數據的完整性。 2.當查詢條件完全相同且包含主鍵，此時用union和union all查詢出來的記 ...
  • Percona TokuDB
    Percona TokuDB Percona TokuDB. 1 1. TokuDB說明... 1 2. TokuDB安裝... 1 3. 使用TokuDB. 1 3.1 快速插入和富索引... 1 3.2 聚集secondary索引... 1 3.3 線上索引創建... 1 3.4 線上添加，刪除 ...
  • 免費簡訊平臺函數說明
    此簡訊平臺，支持法國wavecom簡訊貓.有VB及DELPHI示常式序 錯誤說明0 成功-1 連接簡訊平臺失敗-2 命令執行失敗-3 無可讀簡訊 function ConnectToServer(aServerName, aLogID, aPass: pchar): integer; stdcall ...
  • windows系統下的redis啟動教程
    下載解壓後配置redis.conf文件配置埠號和密碼，打開poweshell命令，進入redis解壓目錄，使用.\redis-server.exe redis.conf 命令啟動redis服務，再打開一個powershell命令，通過命令.\redis-cli -h 127.0.0.1 -p 63 ...
  • C++使用Mysql的詳細步驟及各個常用方法的代碼演示：select,insert,update,delete
    這幾天一直在學習C++下使用Mysql的方法及其中各種的問題，也看了很多Mysql的API函數，當然自己看的還是很基礎的。其實對於每種資料庫的操作，基本的方法都是非常類似的，大多都是connect,select,update,delete以及insert這幾個操作。接下來我就將這幾個步驟以代碼的方式 ...
  • Redis集群~StackExchange.redis連接Sentinel伺服器並訂閱相關事件（原創）
    回到目錄 對於redis-sentinel我在之前的文章中已經說過，它是一個仲裁者，當主master掛了後，它將在所有slave伺服器中進行選舉，選舉的原則當然可以看它的官方文章，這與我們使用者沒有什麼關係，而對於sentinel來說，它在進行主從切換時，會觸發相關事件，這是和我們開發人員有關係的， ...
  • sql-索引的作用(超詳細)
    （一）深入淺出理解索引結構 實際上，您可以把索引理解為一種特殊的目錄。微軟的SQL SERVER提供了兩種索引：聚集索引（clustered index，也稱聚類索引、簇集索引）和非聚集索引（nonclustered index，也稱非聚類索引、非簇集索引）。下麵，我們舉例來說明一下聚集索引和非聚集 ...
一周排行
    -Advertisement-
    Play Games
  • 移動開發（一）：使用.NET MAUI開發第一個安卓APP
    移動開發（一）：使用.NET MAUI開發第一個安卓APP 對於工作多年的C#程式員來說，近來想嘗試開發一款安卓APP，考慮了很久最終選擇使用.NET MAUI這個微軟官方的框架來嘗試體驗開發安卓APP，畢竟是使用Visual Studio開發工具，使用起來也比較的順手，結合微軟官方的教程進行了安卓 ...
  • wpf ToggleButton選中效果和一個登錄界面
    前言 QuestPDF 是一個開源 .NET 庫，用於生成 PDF 文檔。使用了C# Fluent API方式可簡化開發、減少錯誤並提高工作效率。利用它可以輕鬆生成 PDF 報告、發票、導出文件等。 項目介紹 QuestPDF 是一個革命性的開源 .NET 庫，它徹底改變了我們生成 PDF 文檔的方 ...
  • 彈幕樹洞項目功能新增篇
    項目地址 項目後端地址： https://github.com/ZyPLJ/ZYTteeHole 項目前端頁面地址： ZyPLJ/TreeHoleVue (github.com) https://github.com/ZyPLJ/TreeHoleVue 目前項目測試訪問地址： http://tree ...
  • 第27篇 sqlserver2022詳細安裝步驟
    話不多說，直接開乾 一.下載 1.官方鏈接下載： https://www.microsoft.com/zh-cn/sql-server/sql-server-downloads 2.在下載目錄中找到下麵這個小的安裝包 SQL2022-SSEI-Dev.exe，運行開始下載SQL server； 二. ...
  • .NET 開源高性能 MQTT 類庫
    前言 隨著物聯網（IoT）技術的迅猛發展，MQTT（消息隊列遙測傳輸）協議憑藉其輕量級和高效性，已成為眾多物聯網應用的首選通信標準。 MQTTnet 作為一個高性能的 .NET 開源庫，為 .NET 平臺上的 MQTT 客戶端與伺服器開發提供了強大的支持。 本文將全面介紹 MQTTnet 的核心功能 ...
  • Serilog文檔翻譯系列（六） - 可用的接收器、增強器、格式化輸出
    Serilog支持多種接收器用於日誌存儲，增強器用於添加屬性，LogContext管理動態屬性，支持多種輸出格式包括純文本、JSON及ExpressionTemplate。還提供了自定義格式化選項，適用於不同需求。 ...
  • 警惕 Visual Studio 屬性求值副作用導致邏輯不符合預期
    目錄簡介獲取 HTML 文檔解析 HTML 文檔測試參考文章 簡介 動態內容網站使用 JavaScript 腳本動態檢索和渲染數據，爬取信息時需要模擬瀏覽器行為，否則獲取到的源碼基本是空的。 本文使用的爬取步驟如下： 使用 Selenium 獲取渲染後的 HTML 文檔 使用 HtmlAgility ...
  • [使用目前最新版]HybridCLR6.9.0+YooAsset2.2.4實現純C# Unity熱更新方案 (一)
    1.前言 什麼是熱更新 游戲或者軟體更新時，無需重新下載客戶端進行安裝，而是在應用程式啟動的情況下，在內部進行資源或者代碼更新 Unity目前常用熱更新解決方案 HybridCLR，Xlua，ILRuntime等 Unity目前常用資源管理解決方案 AssetBundles，Addressable， ...
  • 在 ASP.NET Core Web API 中使用操作篩選器統一處理通用操作
    本文章主要是在C# ASP.NET Core Web API框架實現向手機發送驗證碼簡訊功能。這裡我選擇是一個互億無線簡訊驗證碼平臺，其實像阿裡雲，騰訊雲上面也可以。 首先我們先去 互億無線 https://www.ihuyi.com/api/sms.html 去註冊一個賬號 註冊完成賬號後，它會送 ...
  • 第28篇 如何.net中實現高效可靠數據同步api
    通過以下方式可以高效，並保證數據同步的可靠性 1.API設計 使用RESTful設計，確保API端點明確，並使用適當的HTTP方法（如POST用於創建，PUT用於更新）。 設計清晰的請求和響應模型，以確保客戶端能夠理解預期格式。 2.數據驗證 在伺服器端進行嚴格的數據驗證，確保接收到的數據符合預期格 ...
所有分類