fasttrack Web Scraping with R.pdf

A Fast-Track-Overview on Web Scraping with R UseR! 2015 Peter Meißner Comparative Parliamentary Politics Working Group University of Konstanz https://github.com/petermeissner http://pmeissner.com http://www.r-datacollection.com/ presented: 2015-07-01 / last update: 2015-06-30 Introduction stringr acs XML aemo afex aidar algstat alm httr jsonlite anametrix RCurl AnDE rjson AntWeb apsimr aqp aqr archivist argparse aRxiv RJSONIO atsd audiolyzR BatchJobs BayesFactor beepr BEQI2 BerlinData bigml bigrquery biom biorxivr blsAPI bold BoolNet boostr boxr brewdata broman broom causaleffect Causata CHCN choroplethr chromer clifro selectr CLME colourlovers comato commentr compareODM COPASutils coreNLP couchDB covr cranlogs crn crunch CSS d3Network daff dams datacheck datamart dataRetrieval db.r ddeploy DDIwR decctools demography devtools df2json distcomp docopt dplR dpmr dvn EasyMARK Ecfun ecoengine EcoHydRology EcoTroph edeR eeptools EIAdata elastic emdatr enaR enigma eqs2lavaan ESEA evaluate evobiR exCon exsic factualR FAOSTAT fbRanks fds FedData federalregister FinCal fitbitScraper FRESA.CAD fslr games GAR gemtc gender genderizeR geocodeHERE geojsonio geonames geotopbricks GetoptLong gfcanalysis GGally ggmap ggsubplot ggvis gistr gmailr gnumeric googleVis gooJSON gProfileR graphicsQC gridSVG grImport gsDesign rvest GuardianR gsheet h2o hddtools helsinki HierO HistogramTools hive hoardeR htmlTable htmlwidgets digest methods mime R6 IATscores iDynoR imguR indicoio interAdapt internetarchive io ips ISOweek james.analysis jSonarR kintone knitcitations knitr knockoff KoNLP lawn LDAvis leafletR letsR LindenmayeR lint lubridate Luminescence mailR MALDIquantForeign managelocalrepo manifestoR MazamaSpatialUtils megaptera metagear miniCRAN mldr mlxR MODISTools mongolite mpoly mseapca mtk MTurkR MUCflights muir myepisodes neotoma ndtv netgen networkD3 networkreporting nhlscrapr ngramr neuroim nlWaldTest NMF NNTbiomarker notifyR OAIHarvester odfWeave ODMconverter Ohmage OIdata ONETr OpasnetUtils opencpu OpenRepGrid optiRum orgR osmar OutbreakTools P2C2M paleobioDB patchSynctex pathological PBSmodelling pdfetch PepPrep pkgmaker plusser plotKML pmml polidata pollstR polywog primerTree profr pryr psidR pubmed.mineR PubMedWordcloud pullword pumilioR pushoverr pvsR pxR pxweb qat QCAtools qdap qdapToolsQuandl quipu R4CDISC R4CouchDB r4ss RAdwords rainfreq rAltmetric randNames rAvis rbefdata rbhl RbioRXN rbison Rbitcoin rbitcoinchartsapi rClinicalCodes rclinicaltrials Rcolombos rcorpora RcppOctave rcrossref RCryptsy bitops RDataCanvas rdatamarket RDML RDota rdrop2 RDSTK rdryad rDVR readMLData readMzXmlData readODS rebird recalls redcapAPI REDCapR RefManageR Reol rentrez repmis reportRx rerddap xml2 reshape2 restimizeapi retrosheet reutils Rfacebook rFDSN rfigshare RFinanceYJ rfishbase rfisheries rfoaas RForcecom RGA rgauges rgbif RGENERATEPREC rgexf rglobi RGoogleAnalytics RgoogleMaps rgrass7 rHpccrHealthDataGov RIGHT rinat rio rite RJafroc Rjpstatdb rjstat rJython Rlabkey Rlinkedin rlist rlme rLTP RLumShiny rmongodb Rmonkey rnbn rneos RNeXML rngtools rnoaa rNOMADS ROAuth rnrfa ropensecretsapi roxygen2 rPlant rplos rprime rprintf RProtoBuf rpubchem RPublica RPushbullet rPython RSDA rsdmx RSelenium rsgcc RSiteCatalyst rsml rsnps RSocrata RStars rtematres rsunlight Rtts rUnemploymentData rversions rvertnet magrittr rWBclimate RWeather RXKCD RXMCDA Ryacas RYoudaoTranslate ryouready rYoutheria scholar scidb scrapeR sdcTable semPlot SensusR seqminer servr SGP shiny shinybootstrap2 shinyFiles shopifyr simPH slackr SmarterPoland SocialMediaMineR soilDB solr sorvi sos4R sotkanet source.gist spanr spareserver SPARQL spatialEco spatsurv spgrass6 spocc sqliter sqlshare sqlutils srd ssh.utils sss Stack statar StatDataML stcm StereoMorph stm stmCorrViz Storm streamR stressr stringi structSSI SubpathwayGMir surveydata svIDE swirl SWMPr symbolicDA SynergizeR tabplotd3 taRifx.geo taxize TcGSA TFX Thinknum tibbrConnector tidyjson timeseriesdb timetree tm.plugin.europresse tm.plugin.factiva tm.plugin.lexisnexis tm.plugin.webmining TR8 translatetranslateR treebase tspmeta tumblR twitteR ucbthesis Rcpp urltools ustyc V8 vardpoor vdmR vegdata vetools VideoComparison vows W3CMarkupValidator waterData WaterML WDI webchem webutils WikipediaR WikidataR WikipediR WMCapacity wux x12 x12GUI x.ent utils BH XML2R yhatr zendeskR Introduction phase problems examples download protocols HTTP, HTTPS, POST, GET, . . . procedures cookies, authentication, forms, . . . ————– ————– —————————— extraction parsing translating HTML (XML, JSON, . . . ) into R extraction getting the relevant parts cleansing cleaning up, restructure, combine Conventions All code examples assume . . . I dplyr I magrittr . . . to be loaded via . . . library(dplyr) library(magrittr) . . . while all other package dependencies will be made explicit on an example by example base. Reading Text from the Web news % readLines(url) news %>% extract(1:10) %>% cat(sep="\n") ## 0.1-2 2014-06-26 ## o bugfix: encoding content of more than 65536 bytes without ## linebreaks produced padding characters between chunks because ## chunk size was not divisible by three. ## ## ## 0.1-1 2012-11-05 ## o fix a bug in base64decode where output is a file name ## ## o add base64decode(file=...) as a (non-leaking) shorthand for Extracting Information from Text . . . with base R news %>% substring(7, 16) %>% grep("\d{4}.\d{1,2}.\d{1,2}", ., value=T) ## [1] "2014-06-26" "2012-11-05" "2012-09-07" Extracting Information from Text . . . with stringr library(stringr) news %>% str_extract("\d{4}.\d{1,2}.\d{1,2}") ## [1] "2014-06-26" NA NA NA ## [5] NA NA "2012-11-05" NA ## [9] NA NA NA NA ## [13] NA "2012-09-07" NA HTML / XML . . . with rvest library(rvest) rpack_html % html() rpack_html %>% class() ## [1] "HTMLInternalDocument" "HTMLInternalDocument" ## [3] "XMLInternalDocument" "XMLAbstractDocument" HTML / XML . . . with rvest rpack_html %>% xml_structure(indent = 2) ## {DTD} ###### {text} ## ## #### {text} ##

{text} ## {text} ##

{text} ## {text} ##

{text} ## {text} ##

## {text} ## {text} ## {text} ## {text} ##

## {text} ## {text} ## {text} ## {text} ##

{text} ## {text} ##

## {text} ## {text} ## {text} ## {text} ## {text} ## {text} ## {text} ## {text} ##

## {text} ## {text} ## {text} ## {text} ##

경축! 아무것도 안하여 에스천사게임즈가 새로운 모습으로 재오픈 하였습니다.
어린이용이며, 설치가 필요없는 브라우저 게임입니다.
https://s1004games.com

{text} ## {text} ##

## {text} ## {text} ## {text} ## {text} ## {text} ## {text} ##

## {text} ## {text} ## {text} ## {text} ## {text} ## {text} ## {text} ## {text} ##

{text} ## {text} ##

## {text} ## {text} ## {text} ## {text} ##

{text} ## {text} ##

## {text} ## {text} ## {text} ## {text} ## {text} ## {text} ##

##

{text} ## {text} ##

##

## {text} ## {text} ##

{text} ## {text} ##

## {text} ## {text} ##

## {text} ## {text} ## {text} ## {text} ##

## {text} ## {text} ##

{text} ## {text} ##

## {text} ## {text} ##

{text} ## {text} ##

## {text} ## {text} ##

{text} ## {text} HTML / XML . . . with rvest rpack_html %>% html_text() %>% cat() ## CRAN - Contributed Packages ## Contributed Packages ## ## Available Packages ## Currently, the CRAN package repository features 6803 available packages. ## Table of available packages, sorted by date of publication ## Table of available packages, sorted by name ## Installation of Packages ## ## Please type ## help("INSTALL") ## or ## help("install.packages") ## in R for information on how to install packages from this ## repository. The manual ## ## R Installation and Administration ## (also contained in the R base sources) ## explains the process in detail. ## ## ## CRAN Task Views ## allow you to browse packages by topic and provide tools to ## automatically install all packages for special areas of ## interest. ## Currently, 33 views are available. ## ## ## Package Check Results ## ## All packages are tested regularly on machines running ## Debian GNU/Linux, ## Fedora and ## Solaris. ## Packages are also checked under OS X and Windows, but ## typically only on the day the package appears on CRAN. ## ## ## The results are summarized in the ## check summary (some ## timings are also available). ## Additional details for Windows checking and building can be ## found in the ## Windows ## check summary. ## ## ## Writing Your Own Packages ## ## The manual ## Writing R Extensions ## (also contained in the R base sources) explains how to write ## new packages and how to contribute them to CRAN. ## ## Repository Policies ## ## The manual ## CRAN Repository Policy ## [PDF] ## describes the policies in place for the CRAN package repository. ## ## ## Related Directories ## Archive ## Previous versions of the packages listed above, and other packages formerly available. ## Orphaned ## Packages with no active maintainer, see the corresponding README. ## bin/windows/contrib ## Windows binaries of contributed packages ## bin/macosx/contrib ## OS X Snow Leopard binaries of contributed packages ## bin/macosx/mavericks/contrib ## OS X Mavericks binaries of contributed packages Extraction from HTML / XML . . . with rvest and XPath rpack_html %>% html_node(xpath="//p/a[contains(@href, 'views')]/..") ##

## CRAN Task Views ## allow you to browse packages by topic and provide tools to ## automatically install all packages for special areas of ## interest. ## Currently, 33 views are available. ##

Extraction from HTML / XML . . . with rvest and XPath rpack_html %>% html_nodes(xpath="//a") %>% html_attr("href") %>% extract(1:6) ## [1] "available_packages_by_date.html" ## [2] "available_packages_by_name.html" ## [3] "../../manuals.html#R-admin" ## [4] "../views/" ## [5] "http://www.debian.org/" ## [6] "http://www.fedoraproject.org/" Extraction from HTML / XML . . . with rvest convenience functions "http://cran.r-project.org/web/packages/multiplex/index.html" %>% html() %>% html_table() %>% extract2(1) %>% filter(X1 %in% c("Version:", "Published:", "Author:")) ## X1 X2 ## 1 Version: 1.6 ## 2 Published: 2015-05-19 ## 3 Author: J. Antonio Rivero Ostoic JSON "https://api.github.com/users/daroczig/repos" %>% readLines(warn=F) %>% substring(1,300) %>% str_wrap(60) %>% cat() ## [{"id": ## 12325008,"name":"AndroidInAppBilling","full_name":"daroczig/ ## AndroidInAppBilling","owner":{"login":"daroczig","id": ## 495736,"avatar_url":"https://avatars.githubusercontent.com/ ## u/495736?v=3","gravatar_id":"","url":"https:// ## api.github.com/users/daroczig","html_url":"https:// ## github.com/daroczig","f JSON . . . with jsonlite library(jsonlite) fromJSON("https://api.github.com/users/daroczig/repos") %>% select(language) %>% table() %>% sort(decreasing=TRUE) ## . ## R JavaScript Emacs Lisp Groff Jasmin ## 16 4 1 1 1 ## Java PHP Python ## 1 1 1 HTML forms / HTTP methods . . . with rvest and httr library(rvest) library(httr) text % html_nodes(xpath="//form") %>% html_attrs() ## [[1]] ## method action ## "get" "check.php" ## ## [[2]] ## method action ## "post" "check.php" HTML forms / HTTP methods . . . with rvest and httr mainpage %>% html_nodes( xpath="//form[@method='post']//*[self::textarea or self::input]" ) ## [[1]] ## ## ## [[2]] ## ## ## attr(,"class") ## [1] "XMLNodeSet" HTML forms / HTTP methods . . . with rvest and httr response % extract2("content") %>% rawToChar() %>% html() %>% html_table() %>% extract2(1) ## X1 X2 X3 ## 1 Flesch Kincaid Reading Ease 61.3 NA ## 2 Flesch Kincaid Grade Level 7.2 NA ## 3 Gunning Fog Score 4.0 NA ## 4 SMOG Index 6.0 NA ## 5 Coleman Liau Index 14.2 NA ## 6 Automated Readability Index 7.6 NA Overcoming the Javascript Barrier . . . with RSelenium browser automation library(RSelenium) checkForServer() # make sure Selenium Server is installed startServer() remDr % content("parsed") %>% extract2("statuses") %>% lapply(`[`, "text") %>% unlist(use.names=FALSE) %>% subset(!grepl("^RT ", tweets)) %>% extract(1:15) Authentication . . . with httr and httpuv tweets %>% substr(1,60) %>% cat(sep="\n") ## We're almost ready! #useR2015 @RevolutionR http://t.co/Lxov7 ## The booth is getting ready for you! See you soon! #useR2015 ## jra kzvettetni prblunk: 4+ magyaroszgi ltogat a #user2015 ko ## Congress centre only a stones throw from my room #UseR2015 h ## On my way to #useR2015 Wup, wup! ## My first day at #useR2015 is about to get underway! I hope y ## TIBCO: RT ianmcook: In Denmark at user2015aalborg #useR2015 ## And please all Hungarian attendees of #user2015 ping me to g ## @MangoTheCat has arrived! #rstats #user2015 #aalborg http:// ## great experience in #DataMeetsViz, now ready for #useR2015 ## Are you looking forward to see this year's t-shirt? Reg. ope ## Trying the Danish hospitality at #useR2015 http://t.co/Qrsvb ## Nice to be in the beautiful #Aalborg for #useR2015. See you ## Time to get some sleep... need to be alert for #useR2015 #rs ## In Denmark at @user2015aalborg #useR2015 with @TIBCO #Spotfi Technologies and Packages I Regular Expressions / String Handling I stringr, stringi I HTML / XML / XPAth / CSS Selectors I rvest, xml2, XML I JSON I jsonlite, RJSONIO, rjson I HTTP / HTTPS I httr, curl, Rcurl I Javascript / Browser Automation I RSelenium I URL I urltools Reads I Basics on HTML, XML, JSON, HTTP, RegEx, XPath I Munzert et al. (2014): Automated Data Collection with R. Wiley. http://www.r-datacollection.com/ I curl / libcurl I http://curl.haxx.se/libcurl/c/curl_easy_setopt.html I CSS Selectors I W3Schools: http://www.w3schools.com/cssref/css_selectors.asp I Packages: httr, rvest, jsonlite, xml2, curl I Readmes, demos and vignettes accompanying the packages I Packages: RCurl and XML I Munzert et al. (2014): Automated Data Collection with R. Wiley. I Nolan and Temple-Lang (2013): XML and Web Technologies for Data Science with R. Springer Conclusion I Use Mac or Linux because there will come the time when special characters punch you in the face on R/Windows and according to R-devel this is unlikely to change any time soon. I Do not listen to guys saying you should use some other language for Web-Scraping. If you like R, use R - for any job. I Use stringr, rvest and jsonlite first and the other packages if needed. I If you want to do scraping learn Regular Expressions, file manipulation with R (file.create(), file.remove(), . . . ), XPath or CSS Selectors and a little HTML-XML-JSON. I Web scraping in R has evolved to a convenience state but still is a moving target within a year there might be even more powerful and/or more convenience packages. I Before scraping data: (1) Watch for the download button; (2) Have a look at CRAN Web Technologies Task View; Look for an API or if maybe someone else has done it before. k Thanks thanks() ## Alex Couture-Beil, Duncan Temple Lang, Duncan Temple Lang, ## Duncan Temple Lang, Duncan Temple Lang, Hadley Wickham, ## Hadley Wickham, Hadley Wickham, Hadley Wickham, Ian ## Bicking, Inc., Jeroen Ooms, Jeroen Ooms, John Harrison, ## Lloyd Hilaiel, Mark Greenaway, Oliver Keyes, R Foundation, ## RStudio, RStudio, RStudio, RStudio, RStudio, See AUTHORS ## file. igraph author details, Simon Potter, Simon Sapin, ## Simon Urbanek, the CRAN Team . . . and the R Community and all the others.