Web-Scraping-with-R-XiaoNan.pdf

. Web Scra . �in� wit� R Xiao Nan @road2stat 6th China R Beijing Xiao Nan @road2stat Web Scraping with R 1/43 . 1/43 . Outline • Overview • Toolkit • Exception Handling • Parallelization • Outro Xiao Nan @road2stat Web Scraping with R 2/43 . 2/43 . Part 1 Overview Two Types of Scrapers / Crawlers The REAL ones and the ... . Fake ones? . I’m not a fake! I can crawl the web, too! . Two Types of Scrapers • General-purpose Crawlers • Focused Crawlers (our focus today) Xiao Nan @road2stat Web Scraping with R 8/43 . 8/43 . Browser Revisited . Xiao Nan @road2stat Web Scraping with R 9/43 . 9/43 . Browser Revisited . Analysis and Retrieve Xiao Nan @road2stat Web Scraping with R 10/43 . 10/43 . Browser Revisited . Analysis and Retrieve Receive and Parse Xiao Nan @road2stat Web Scraping with R 11/43 . 11/43 . Comparing to Other Languages Pros & Cons . Pros . . • Lightweight • Easy to implement • Easy to debug • Seamless modeling integration: less I/O Xiao Nan @road2stat Web Scraping with R 12/43 . 12/43 . Comparing to Other Languages Pros & Cons . Pros . . • Lightweight • Easy to implement • Easy to debug • Seamless modeling integration: less I/O . Cons . . • Fewer libraries (than Python & Ruby) • Multi-Process Parallelization: forking is deàcient ... Xiao Nan @road2stat Web Scraping with R 13/43 . 13/43 . Choosing A Better Platform . Xiao Nan @road2stat Web Scraping with R 14/43 . 14/43 . Choosing A Better Platform Why Linux? • Network performance & mem. management → Faster • Better parallelization support → Faster • Uniàed encoding & locale → Faster (for coders) • More recent third party libs → Faster (less bugs) Xiao Nan @road2stat Web Scraping with R 15/43 . 15/43 . Choosing A Better Platform Why Linux? • Network performance & mem. management → Faster • Better parallelization support → Faster • Uniàed encoding & locale → Faster (for coders) • More recent third party libs → Faster (less bugs) Xiao Nan @road2stat Web Scraping with R 15/43 . 15/43 . Choosing A Better Platform Why Linux? • Network performance & mem. management → Faster • Better parallelization support → Faster • Uniàed encoding & locale → Faster (for coders) • More recent third party libs → Faster (less bugs) Xiao Nan @road2stat Web Scraping with R 15/43 . 15/43 . Choosing A Better Platform Why Linux? • Network performance & mem. management → Faster • Better parallelization support → Faster • Uniàed encoding & locale → Faster (for coders) • More recent third party libs → Faster (less bugs) Xiao Nan @road2stat Web Scraping with R 15/43 . 15/43 . Choosing A Better Platform Why Linux? • Network performance & mem. management → Faster • Better parallelization support → Faster • Uniàed encoding & locale → Faster (for coders) • More recent third party libs → Faster (less bugs) Xiao Nan @road2stat Web Scraping with R 15/43 . 15/43 . Part 2 Toolkit Retrieve & Parse . Toolkit Available R Packages Pkg. Name Retrieve? Parse? Must-Know? RCurl Yes No Yes XML Limited Yes Yes rjson No Yes Yes RJSONIO No Yes Optional httr Yes Yes Optional selectr No Yes Optional ROAuth No No Optional Xiao Nan @road2stat Web Scraping with R 18/43 . 18/43 . Toolkit Available R Packages . RCurl [XML] . XML . selectr . ROAuth . httr . [JSON] rjson . . RJSONIO . . Parsing Retrieving Lightweight Heavyweight Xiao Nan @road2stat Web Scraping with R 19/43 . 19/43 . Toolkit Available R Packages • RCurl → Header Conàguration R 不务正业之 RCurl: http://cos.name/cn/topic/17816 • XML → XPath, 3 Critical Functions http://www.road2stat.com/cn/r/rxml.html • rjson → Ofàcially Listed / RJSONIO → by D.T.L. • httr → Simpliàcation version RCurl + XML + rjson Not Recommended for not discreet enough: http://randyzwitch.com/r-error-message-fun/ • ROAuth → Useful for APIs. see RWeibo of @lijian001 • selectr → Translate CSS Selectors to XPath Expressions Xiao Nan @road2stat Web Scraping with R 20/43 . 20/43 . Toolkit Available R Packages • RCurl → Header Conàguration R 不务正业之 RCurl: http://cos.name/cn/topic/17816 • XML → XPath, 3 Critical Functions http://www.road2stat.com/cn/r/rxml.html • rjson → Ofàcially Listed / RJSONIO → by D.T.L. • httr → Simpliàcation version RCurl + XML + rjson Not Recommended for not discreet enough: http://randyzwitch.com/r-error-message-fun/ • ROAuth → Useful for APIs. see RWeibo of @lijian001 • selectr → Translate CSS Selectors to XPath Expressions Xiao Nan @road2stat Web Scraping with R 20/43 . 20/43 . Toolkit Available R Packages • RCurl → Header Conàguration R 不务正业之 RCurl: http://cos.name/cn/topic/17816 • XML → XPath, 3 Critical Functions http://www.road2stat.com/cn/r/rxml.html • rjson → Ofàcially Listed / RJSONIO → by D.T.L. • httr → Simpliàcation version RCurl + XML + rjson Not Recommended for not discreet enough: http://randyzwitch.com/r-error-message-fun/ • ROAuth → Useful for APIs. see RWeibo of @lijian001 • selectr → Translate CSS Selectors to XPath Expressions Xiao Nan @road2stat Web Scraping with R 20/43 . 20/43 . Toolkit Available R Packages • RCurl → Header Conàguration R 不务正业之 RCurl: http://cos.name/cn/topic/17816 • XML → XPath, 3 Critical Functions http://www.road2stat.com/cn/r/rxml.html • rjson → Ofàcially Listed / RJSONIO → by D.T.L. • httr → Simpliàcation version RCurl + XML + rjson Not Recommended for not discreet enough: http://randyzwitch.com/r-error-message-fun/ • ROAuth → Useful for APIs. see RWeibo of @lijian001 • selectr → Translate CSS Selectors to XPath Expressions Xiao Nan @road2stat Web Scraping with R 20/43 . 20/43 . Toolkit Available R Packages • RCurl → Header Conàguration R 不务正业之 RCurl: http://cos.name/cn/topic/17816 • XML → XPath, 3 Critical Functions http://www.road2stat.com/cn/r/rxml.html • rjson → Ofàcially Listed / RJSONIO → by D.T.L. • httr → Simpliàcation version RCurl + XML + rjson Not Recommended for not discreet enough: http://randyzwitch.com/r-error-message-fun/ • ROAuth → Useful for APIs. see RWeibo of @lijian001 • selectr → Translate CSS Selectors to XPath Expressions Xiao Nan @road2stat Web Scraping with R 20/43 . 20/43 . Toolkit Available R Packages • RCurl → Header Conàguration R 不务正业之 RCurl: http://cos.name/cn/topic/17816 • XML → XPath, 3 Critical Functions http://www.road2stat.com/cn/r/rxml.html • rjson → Ofàcially Listed / RJSONIO → by D.T.L. • httr → Simpliàcation version RCurl + XML + rjson Not Recommended for not discreet enough: http://randyzwitch.com/r-error-message-fun/ • ROAuth → Useful for APIs. see RWeibo of @lijian001 • selectr → Translate CSS Selectors to XPath Expressions Xiao Nan @road2stat Web Scraping with R 20/43 . 20/43 . Toolkit Front-End and Miscellaneous • Chrome Developer Tools / FireBug → Analyzing AJAX Requests: http://cos.name/cn/topic/107729 • JSONView → Output Formatted JSON • Visual Event → Bounded event on DOM elements • tcpdump + Wireshark → Packet Capture & Protocol Analysis Xiao Nan @road2stat Web Scraping with R 21/43 . 21/43 . Toolkit Front-End and Miscellaneous • Chrome Developer Tools / FireBug → Analyzing AJAX Requests: http://cos.name/cn/topic/107729 • JSONView → Output Formatted JSON • Visual Event → Bounded event on DOM elements • tcpdump + Wireshark → Packet Capture & Protocol Analysis Xiao Nan @road2stat Web Scraping with R 21/43 . 21/43 . Toolkit Front-End and Miscellaneous • Chrome Developer Tools / FireBug → Analyzing AJAX Requests: http://cos.name/cn/topic/107729 • JSONView → Output Formatted JSON • Visual Event → Bounded event on DOM elements • tcpdump + Wireshark → Packet Capture & Protocol Analysis Xiao Nan @road2stat Web Scraping with R 21/43 . 21/43 . Toolkit Front-End and Miscellaneous • Chrome Developer Tools / FireBug → Analyzing AJAX Requests: http://cos.name/cn/topic/107729 • JSONView → Output Formatted JSON • Visual Event → Bounded event on DOM elements • tcpdump + Wireshark → Packet Capture & Protocol Analysis Xiao Nan @road2stat Web Scraping with R 21/43 . 21/43 . Part 3 Exception Handling . More than 70% . Exception Handling Coding Strategy • Dirty HTML & XML: Preprocess with htmltidy • Build-in Condition/Error Handler Function: XML::xmlStructuredStop • Coding Strategy: Interative until fault-tolerant. Xiao Nan @road2stat Web Scraping with R 25/43 . 25/43 . Exception Handling Coding Strategy • Dirty HTML & XML: Preprocess with htmltidy • Build-in Condition/Error Handler Function: XML::xmlStructuredStop • Coding Strategy: Interative until fault-tolerant. Xiao Nan @road2stat Web Scraping with R 25/43 . 25/43 . Exception Handling Coding Strategy • Dirty HTML & XML: Preprocess with htmltidy • Build-in Condition/Error Handler Function: XML::xmlStructuredStop • Coding Strategy: Interative until fault-tolerant. Xiao Nan @road2stat Web Scraping with R 25/43 . 25/43 . Exception Handling FAQ on COS BBS • Cookie Operation → http://cos.name/cn/topic/108806 • Referer Validation → http://cos.name/cn/topic/109407 • Session Validation → http://cos.name/cn/topic/107802 • Encoding Errors → Identify the Problem Source Google: 编码 site:cos.name/cn/ Xiao Nan @road2stat Web Scraping with R 26/43 . 26/43 . Exception Handling The Various Data Source • Choose Ofàcial API àrst: NCBI with rOpenSci • Restricted API usage: Private API key Xiao Nan @road2stat Web Scraping with R 27/43 . 27/43 . Exception Handling The Various Data Source • Choose Ofàcial API àrst: NCBI with rOpenSci • Restricted API usage: Private API key Xiao Nan @road2stat Web Scraping with R 27/43 . 27/43 . . Exception Handling The Various Data Source • Choose Ofàcial API àrst (NCBI with rOpenSci) • Restricted API usage: Private API key • SSL and SSL Decryption: Trusted MITM http://www.webos-internals.org/wiki/Decrypt_SSL_(trusted_ man-in-the-middle_technique) Xiao Nan @road2stat Web Scraping with R 29/43 . 29/43 . Part 4 Parallelization . . Parallelization The best solution is? A Conventional Way: RCurl::getURIAsynchronous() • Native. Extremely easy to use. • Pitfalls: Have to control the process number by hand. This seems weird! Xiao Nan @road2stat Web Scraping with R 32/43 . 32/43 . Parallelization The best solution is? A Conventional Way: RCurl::getURIAsynchronous() • Native. Extremely easy to use. • Pitfalls: Have to control the process number by hand. This seems weird! Xiao Nan @road2stat Web Scraping with R 32/43 . 32/43 . Parallelization The best solution is? A Conventional Way: RCurl::getURIAsynchronous() • Native. Extremely easy to use. • Pitfalls: Have to control the process number by hand. This seems weird! Xiao Nan @road2stat Web Scraping with R 32/43 . 32/43 . Parallelization The best solution is? A Better Way: doMC + foreach • Full control / Easy to migrate / Natural to code: require(doMC) registerDoMC(20) x <- foreach(i = 1:1e+5, ...) %dopar% { xxx <- getURL(urls[i]) } Xiao Nan @road2stat Web Scraping with R 33/43 . 33/43 . Parallelization The best solution is? A Better Way: doMC + foreach • Full control / Easy to migrate / Natural to code: require(doMC) registerDoMC(20) x <- foreach(i = 1:1e+5, ...) %dopar% { xxx <- getURL(urls[i]) } Xiao Nan @road2stat Web Scraping with R 34/43 . 34/43 . Parallelization The best solution is? A Better Way: doMC + foreach • Full control / Easy to migrate / Natural to code: require(doMC) registerDoMC(20) x <- foreach(i = 1:1e+5, ...) %dopar% { xxx <- getURL(urls[i]) } • Single machine, registerDoMC(20), 10 min, 1e+5 pages. Xiao Nan @road2stat Web Scraping with R 35/43 . 35/43 . Parallelization The best solution is? A Better Way: doMC + foreach • Full control / Easy to migrate / Natural to code: require(doMC) registerDoMC(20) x <- foreach(i = 1:1e+5, ...) %dopar% { xxx <- getURL(urls[i]) } • Single machine, registerDoMC(20), 10 min, 1e+5 pages. • Pitfalls: (Almost) Linux only. Xiao Nan @road2stat Web Scraping with R 36/43 . 36/43 . Parallelization More Pitfalls • Requires high-perf storage → Redis (rredis) or MongoDB (RMongo)? • Memory leak (RCurl & XML) → Avoid long exec. time • Intensive testing before run → Minimize errors Xiao Nan @road2stat Web Scraping with R 37/43 . 37/43 . Part 5 Outro . Remarks Web Crawler Ethics • Web Crawler Ethics • Honor robots.txt • A Balanced Crawling Rate • Spammer Shame • With great power comes great responsibility. Xiao Nan @road2stat Web Scraping with R 39/43 . 39/43 . Remarks Web Crawler Ethics • Web Crawler Ethics • Honor robots.txt • A Balanced Crawling Rate • Spammer Shame • With great power comes great responsibility. Xiao Nan @road2stat Web Scraping with R 39/43 . 39/43 . Remarks Web Crawler Ethics • Web Crawler Ethics • Honor robots.txt • A Balanced Crawling Rate • Spammer Shame • With great power comes great responsibility. Xiao Nan @road2stat Web Scraping with R 39/43 . 39/43 . Remarks Web Crawler Ethics • Web Crawler Ethics • Honor robots.txt • A Balanced Crawling Rate • Spammer Shame • With great power comes great responsibility. Xiao Nan @road2stat Web Scraping with R 39/43 . 39/43 . Remarks Web Crawler Ethics • Web Crawler Ethics • Honor robots.txt • A Balanced Crawling Rate • Spammer Shame • With great power comes great responsibility. Xiao Nan @road2stat Web Scraping with R 39/43 . 39/43 . . Further Reading 1. XML & JSON Speciàcation (esp. XPath) 2. RCurl & XML Documentation 3. Web Data Mining (Chapter 8) by Bing Liu 4. XML and Web Technologies for Data Sciences with R by Duncan Temple Lang, et al. (Due Sep. 2013) 5. Curl.jl https://github.com/forio/Curl.jl Xiao Nan @road2stat Web Scraping with R 41/43 . 41/43 http://cos.name/cn/ Q & A

경축! 아무것도 안하여 에스천사게임즈가 새로운 모습으로 재오픈 하였습니다.
어린이용이며, 설치가 필요없는 브라우저 게임입니다.
https://s1004games.com