##################################################################.##################################################################.#### In this file:#### - Using the rvest package to scrape information from a## basic HTML website#### - Packaging up the code to scrape a page into a function#### - Using a loop to scrape several pages of information## whose URLs differ in a predictable way.#### - Fixing fragile CSS selectors## ## - Intro to regular expressions (regex)#################################################################.#################################################################.#-------------------------------------------------------------------------------# Some online resources## video demonstrating basics of web scraping in R# - https://www.youtube.com/watch?v=v8Yh_4oE-Fs&t=275s## Tips for editing HTML in VSCode# - https://code.visualstudio.com/docs/languages/html## Tips on how to use VSCode in general:# https://code.visualstudio.com/docs/getstarted/tips-and-tricks## timing R code# - https://www.r-bloggers.com/2017/05/5-ways-to-measure-running-time-of-r-code/## video - using RSelenium to scrape dynamic (ie. javascript) webpage# - https://www.youtube.com/watch?v=CN989KER4pA## CSS selector rules# - https://flukeout.github.io/ # - answers to the "game": https://gist.github.com/humbertodias/b878772e823fd9863a1e4c2415a3f7b6## Intro to regular expressions# - https://ryanstutorials.net/regular-expressions-tutorial/#-------------------------------------------------------------------------------# Install (if necessary) and load the rvest package# The following two lines will install and load the rvest package.# I commented out these lines in favor of the one line below##install.packages("rvest") # installs it on your machine (only need to do this once on your machine)#library(rvest) # do this everytime your start RStudio (also require(rvest) works)# This one line will accomplish what the two lines above do. However,# this one line will not install the package if it's not necessary.if(!require(rvest)) install.packages("rvest") # this automatically also loads the "xml2" package
############################################################################# Code to download high/low temperature data from the following website############################################################################url ="https://forecast.weather.gov/MapClick.php?lat=37.7771&lon=-122.4196#.Xl0j6BNKhTY"# You can get a very sepecific CSS selector for a particular # part of a webpage by using the "copy css selector" feature of# your browser. Every browser has a slightly different way of doing this# but they are all similar. In Chrome do the following:## (1) right click on the portion of the page you want to scrape# and choose "inspect"# (2) right click on the portion of the HTML that contains what# you want and choose "Copy" and then "Copy Selector"# # This gives you a very, very specific selector to get this info.# # For exmaple, we followed this process for the first high temperature # for this webpage and got the following selector. ## CSS SELECTOR: (note the leading # on the line below is part of the selector)# #seven-day-forecast-list > li:nth-child(1) > div > p.temp.temp-high## WARNING - this is a VERY specific selector! You will often need# to modify this selector so that it is LESS specific and gets # more than just the one piece of info you clicked on.weatherPage =read_html(url)cssSelector ="#seven-day-forecast-list > li:nth-child(1) > div > p.temp.temp-high"temperature = weatherPage %>%html_elements( cssSelector ) %>%html_text2()temperature
List of 2
$ node:<externalptr>
$ doc :<externalptr>
- attr(*, "class")= chr [1:2] "xml_document" "xml_node"
26.5 Choosing better CSS Selectors
####################################################################.# If we analyze the HTML code, a MUCH BETTER selector is .temp# This selects all html elements that have class="temp"####################################################################.#------------------------------------------------------------------.# NOTE: ## When webscraping you should "play nicely" with the website.# We already read the page above using the following line of code:## weatherPage = read_html(url)## Don't do this again. Every time you read the page you are causing# the website to do some work to generate the page. You also use# network "bandwidth". #------------------------------------------------------------------.# Don't do this again - we already did it. Read comment above for more info.## weatherPage = read_html(url)cssSelector =".temp"forecasts <- weatherPage %>%html_elements( cssSelector ) %>%html_text()forecasts
List of 2
$ node:<externalptr>
$ doc :<externalptr>
- attr(*, "class")= chr [1:2] "xml_document" "xml_node"
26.6 Scraping country names
############################################################################# Get the country names off the following website############################################################################url ="https://scrapethissite.com/pages/simple/"cssSelector =".country-name"countries <-read_html(url) %>%html_elements( cssSelector ) %>%html_text()x =read_html(url)x
#..............................................................................# Let's examine the results.# # Notice that the results includes the newlines (\n) and blanks from the .html# file. This is because we picked up EVERYTHING that appears between the# start tag and end tag in the HTML.## Below, we will see how to fix up the results below by using the gsub function.#..............................................................................countries
############################################################################# same thing a slightly different way (without magrittr pipes)############################################################################url ="https://scrapethissite.com/pages/simple/"cssSelector =".country-name"whole_html_page =read_html(url)country_name_html =html_elements( whole_html_page, cssSelector )just_the_text =html_text(country_name_html)#...........................# Let's examine each part #...........................# the contents of the entire HTML pagewhole_html_page
# remove the actual start and end tags ... leaving just the text.# again ... notice that this picked up the newlines (\n) and blanks from the .html filejust_the_text
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@# Let's look a little closer at the contents of what is returned by read_html#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@# We already had this above, just reviewing it again ...url ="https://scrapethissite.com/pages/simple/"whole_html_page =read_html(url)country_name_html =html_elements( whole_html_page, cssSelector )just_the_text =html_text(country_name_html)# What is the structure of this data?str(whole_html_page) # list of 2 externalptr objects (see below)
List of 2
$ node:<externalptr>
$ doc :<externalptr>
- attr(*, "class")= chr [1:2] "xml_document" "xml_node"
# What is the class of this data? class(whole_html_page) # "xml_document" "xml_node"
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@# An externalptr cannot be accessed directly via R code. It is a "blob"# (a "binary large object") i.e. a piece of data that is only accessible# via C language (not R) code that is used to build some of the packages# that are used to extend R's functionality. You must use the built in# functions in the xml2 and the rvest packages to access this data.#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
26.8 Working with the data you scraped
#.....................................................................## we can eliminate extra newlines and the blanks with the gsub function# #.....................................................................?gsub # see the documentation
[1] " Andorra "
[2] " United Arab Emirates "
[3] " Afghanistan "
[4] " Antigua and Barbuda "
[5] " Anguilla "
[6] " Albania "
[7] " Armenia "
[8] " Angola "
[9] " Antarctica "
[10] " Argentina "
[11] " American Samoa "
[12] " Austria "
[13] " Australia "
[14] " Aruba "
[15] " Åland "
[16] " Azerbaijan "
[17] " Bosnia and Herzegovina "
[18] " Barbados "
[19] " Bangladesh "
[20] " Belgium "
[21] " Burkina Faso "
[22] " Bulgaria "
[23] " Bahrain "
[24] " Burundi "
[25] " Benin "
[26] " Saint Barthélemy "
[27] " Bermuda "
[28] " Brunei "
[29] " Bolivia "
[30] " Bonaire "
[31] " Brazil "
[32] " Bahamas "
[33] " Bhutan "
[34] " Bouvet Island "
[35] " Botswana "
[36] " Belarus "
[37] " Belize "
[38] " Canada "
[39] " Cocos [Keeling] Islands "
[40] " Democratic Republic of the Congo "
[41] " Central African Republic "
[42] " Republic of the Congo "
[43] " Switzerland "
[44] " Ivory Coast "
[45] " Cook Islands "
[46] " Chile "
[47] " Cameroon "
[48] " China "
[49] " Colombia "
[50] " Costa Rica "
[51] " Cuba "
[52] " Cape Verde "
[53] " Curacao "
[54] " Christmas Island "
[55] " Cyprus "
[56] " Czech Republic "
[57] " Germany "
[58] " Djibouti "
[59] " Denmark "
[60] " Dominica "
[61] " Dominican Republic "
[62] " Algeria "
[63] " Ecuador "
[64] " Estonia "
[65] " Egypt "
[66] " Western Sahara "
[67] " Eritrea "
[68] " Spain "
[69] " Ethiopia "
[70] " Finland "
[71] " Fiji "
[72] " Falkland Islands "
[73] " Micronesia "
[74] " Faroe Islands "
[75] " France "
[76] " Gabon "
[77] " United Kingdom "
[78] " Grenada "
[79] " Georgia "
[80] " French Guiana "
[81] " Guernsey "
[82] " Ghana "
[83] " Gibraltar "
[84] " Greenland "
[85] " Gambia "
[86] " Guinea "
[87] " Guadeloupe "
[88] " Equatorial Guinea "
[89] " Greece "
[90] " South Georgia and the South Sandwich Islands "
[91] " Guatemala "
[92] " Guam "
[93] " Guinea-Bissau "
[94] " Guyana "
[95] " Hong Kong "
[96] " Heard Island and McDonald Islands "
[97] " Honduras "
[98] " Croatia "
[99] " Haiti "
[100] " Hungary "
[101] " Indonesia "
[102] " Ireland "
[103] " Israel "
[104] " Isle of Man "
[105] " India "
[106] " British Indian Ocean Territory "
[107] " Iraq "
[108] " Iran "
[109] " Iceland "
[110] " Italy "
[111] " Jersey "
[112] " Jamaica "
[113] " Jordan "
[114] " Japan "
[115] " Kenya "
[116] " Kyrgyzstan "
[117] " Cambodia "
[118] " Kiribati "
[119] " Comoros "
[120] " Saint Kitts and Nevis "
[121] " North Korea "
[122] " South Korea "
[123] " Kuwait "
[124] " Cayman Islands "
[125] " Kazakhstan "
[126] " Laos "
[127] " Lebanon "
[128] " Saint Lucia "
[129] " Liechtenstein "
[130] " Sri Lanka "
[131] " Liberia "
[132] " Lesotho "
[133] " Lithuania "
[134] " Luxembourg "
[135] " Latvia "
[136] " Libya "
[137] " Morocco "
[138] " Monaco "
[139] " Moldova "
[140] " Montenegro "
[141] " Saint Martin "
[142] " Madagascar "
[143] " Marshall Islands "
[144] " Macedonia "
[145] " Mali "
[146] " Myanmar [Burma] "
[147] " Mongolia "
[148] " Macao "
[149] " Northern Mariana Islands "
[150] " Martinique "
[151] " Mauritania "
[152] " Montserrat "
[153] " Malta "
[154] " Mauritius "
[155] " Maldives "
[156] " Malawi "
[157] " Mexico "
[158] " Malaysia "
[159] " Mozambique "
[160] " Namibia "
[161] " New Caledonia "
[162] " Niger "
[163] " Norfolk Island "
[164] " Nigeria "
[165] " Nicaragua "
[166] " Netherlands "
[167] " Norway "
[168] " Nepal "
[169] " Nauru "
[170] " Niue "
[171] " New Zealand "
[172] " Oman "
[173] " Panama "
[174] " Peru "
[175] " French Polynesia "
[176] " Papua New Guinea "
[177] " Philippines "
[178] " Pakistan "
[179] " Poland "
[180] " Saint Pierre and Miquelon "
[181] " Pitcairn Islands "
[182] " Puerto Rico "
[183] " Palestine "
[184] " Portugal "
[185] " Palau "
[186] " Paraguay "
[187] " Qatar "
[188] " Réunion "
[189] " Romania "
[190] " Serbia "
[191] " Russia "
[192] " Rwanda "
[193] " Saudi Arabia "
[194] " Solomon Islands "
[195] " Seychelles "
[196] " Sudan "
[197] " Sweden "
[198] " Singapore "
[199] " Saint Helena "
[200] " Slovenia "
[201] " Svalbard and Jan Mayen "
[202] " Slovakia "
[203] " Sierra Leone "
[204] " San Marino "
[205] " Senegal "
[206] " Somalia "
[207] " Suriname "
[208] " South Sudan "
[209] " São Tomé and Príncipe "
[210] " El Salvador "
[211] " Sint Maarten "
[212] " Syria "
[213] " Swaziland "
[214] " Turks and Caicos Islands "
[215] " Chad "
[216] " French Southern Territories "
[217] " Togo "
[218] " Thailand "
[219] " Tajikistan "
[220] " Tokelau "
[221] " East Timor "
[222] " Turkmenistan "
[223] " Tunisia "
[224] " Tonga "
[225] " Turkey "
[226] " Trinidad and Tobago "
[227] " Tuvalu "
[228] " Taiwan "
[229] " Tanzania "
[230] " Ukraine "
[231] " Uganda "
[232] " U.S. Minor Outlying Islands "
[233] " United States "
[234] " Uruguay "
[235] " Uzbekistan "
[236] " Vatican City "
[237] " Saint Vincent and the Grenadines "
[238] " Venezuela "
[239] " British Virgin Islands "
[240] " U.S. Virgin Islands "
[241] " Vietnam "
[242] " Vanuatu "
[243] " Wallis and Futuna "
[244] " Samoa "
[245] " Kosovo "
[246] " Yemen "
[247] " Mayotte "
[248] " South Africa "
[249] " Zambia "
[250] " Zimbabwe "
# The following has a bug in that it removes ALL spaces.# countries = gsub(" " , "" , without_backslash_n)# Just remove the spaces that appear BEFORE the country namecountries1 =gsub("^ *", "", without_backslash_n)countries2 =gsub(" *$", "", countries1)countries = countries2# alternatively, you can eliminate spaces and \n characters # before and after the text all in one shotcountries3 =gsub("(^[ \\n]*)|([ \\n]*$)", "", without_backslash_n)countries3
########################################################## Now that we understand the basics we can get# all of the country data and put it in a dataframe#########################################################url ="https://scrapethissite.com/pages/simple/"the_full_html =read_html(url)countries = the_full_html %>%html_elements( ".country-name" ) %>%html_text()capitals = the_full_html %>%html_elements( ".country-capital" ) %>%html_text()population = the_full_html %>%html_elements( ".country-population" ) %>%html_text()area = the_full_html %>%html_elements( ".country-area" ) %>%html_text()countries
head(countries,5) # the United Arab Emirates has no spaces! Let's fix that ...
[1] "Andorra" "United Arab Emirates" "Afghanistan"
[4] "Antigua and Barbuda" "Anguilla"
countries = the_full_html %>%html_elements( ".country-name" ) %>%html_text()# remove the spaces and newline characters from countries# Do not remove spaces between wordscountries=gsub("\\n", "", countries)countries=gsub("^ *", "" , countries) # get rid of spaces before the textcountries=gsub(" *$", "" , countries) # get rid of spaces after the textcountries
country capital pop area
1 Andorra Andorra la Vella 84000 468.0
2 United Arab Emirates Abu Dhabi 4975593 82880.0
3 Afghanistan Kabul 29121286 647500.0
4 Antigua and Barbuda St. John's 86754 443.0
5 Anguilla The Valley 13254 102.0
###################################################### UPDATE:## The latest release of the rvest package includes# a function named html_text2. This function automatically# removes leading and trailing whitespace.# It's still important to understand how to modify the # data with gsub for situations for which you want to modify# the data other than leading and trailing whitespace.#####################################################url ="https://scrapethissite.com/pages/simple/"cssSelector =".country-name"whole_html_page =read_html(url)country_name_html =html_elements( whole_html_page, cssSelector )# Using html_text2 instead of html_text.# html_text2 automatically removes leading and trailing whitespace# from the text.just_the_text =html_text2(country_name_html)just_the_text
####################################################### Package up the technique into a function######################################################getCountryData <-function() {# Get all of the country data and put it in a dataframe url ="https://scrapethissite.com/pages/simple/" the_full_html =read_html(url) countries = the_full_html %>%html_elements( ".country-name" ) %>%html_text() capitals = the_full_html %>%html_elements( ".country-capital" ) %>%html_text() population = the_full_html %>%html_elements( ".country-population" ) %>%html_text() area = the_full_html %>%html_elements( ".country-area" ) %>%html_text()# remove the spaces and newline characters from countries countries=gsub("\\n", "", countries) countries=gsub(" ", "" , countries) df =data.frame ( country=countries,capital = capitals,pop = population,area = area,stringsAsFactors =FALSE )return (df)}# Data on websites can change. We can now call this function whenever# we want to get the latest versions of the data from the website.mydata =getCountryData()head(mydata)
country capital pop area
1 Andorra Andorra la Vella 84000 468.0
2 UnitedArabEmirates Abu Dhabi 4975593 82880.0
3 Afghanistan Kabul 29121286 647500.0
4 AntiguaandBarbuda St. John's 86754 443.0
5 Anguilla The Valley 13254 102.0
6 Albania Tirana 2986952 28748.0
26.12 Scraping multiple webpages in a loop
26.12.1 Figure out the code
######################################################## Getting data from multiple web pages in a loop.######################################################## The data on the following page is only one page of multiple pages of similar# data. ## https://scrapethissite.com/pages/forms/?page_num=1## The links to the other pages appear on the bottom of the page. Clicking on # the the link for the 2nd page, reveals that the 2nd page of data is at # the following URL:## https://scrapethissite.com/pages/forms/?page_num=2## These URLs differ ONLY in the page number. This allows us to scrape# ALL of the pages by reconstructing the URL for the page we want by# inserting the correct page number. # CSS FROM INSPECTcss ="#hockey > div > table > tbody > tr:nth-child(2) > td.name"# After looking at the full HTML code, we could see the the tbody# in the selector should not have been there? There could be several # reasons (beyond the scope of today's class) as to why the tbody was # in the css selector that we got back. By looking at the full HTML# and understanding a little about how HTML and css selectors work, we were# able to realize that "tbody" didn't belong and take it out.css ="#hockey > div > table > tr:nth-child(2) > td.name"# When we tried the above css selector we ONLY got back the "Colorado Avalance"# but no other team name. We quickly realized that was because of # :nth-child(2) that was limiting the results. We we took off :nth-child(2)# that got us all of the team names. css ="#hockey > div > table > tr > td.name"# The following is what we got from selector gadgetcss =".name"# according to selector gadget this will match all data in the table# and it does. However, the data comes back in a single vector # instead of a dataframe.css ="td"# Instead of getting all the data in a single vector it's probably better to# get the Name column in one vector and the Year column in another vectorcssName =".name"cssYear =".year"url="https://www.scrapethissite.com/pages/forms/?page_num=7"fullPage =read_html(url)fullPage %>%html_elements(css) %>%html_text2()
# Get the name datateamNames = fullPage %>%html_elements(cssName) %>%html_text2()# Reminder about the %>% pipe symbold# Same as above without using the %>% pipe symbolhtml_text2(html_elements(fullPage, cssName))
[1] "Colorado Avalanche" "Dallas Stars"
[3] "Detroit Red Wings" "Edmonton Oilers"
[5] "Florida Panthers" "Hartford Whalers"
[7] "Los Angeles Kings" "Montreal Canadiens"
[9] "New Jersey Devils" "New York Islanders"
[11] "New York Rangers" "Ottawa Senators"
[13] "Philadelphia Flyers" "Phoenix Coyotes"
[15] "Pittsburgh Penguins" "San Jose Sharks"
[17] "St. Louis Blues" "Tampa Bay Lightning"
[19] "Toronto Maple Leafs" "Vancouver Canucks"
[21] "Washington Capitals" "Mighty Ducks of Anaheim"
[23] "Boston Bruins" "Buffalo Sabres"
[25] "Calgary Flames"
# The %>% symbol takes the ootput of the command on the left and # sends it into the first argument of the command on the right.rep(c(100,200), 3)
[1] 100 200 100 200 100 200
c(100,200) %>%rep(3)
[1] 100 200 100 200 100 200
teamNames
[1] "Colorado Avalanche" "Dallas Stars"
[3] "Detroit Red Wings" "Edmonton Oilers"
[5] "Florida Panthers" "Hartford Whalers"
[7] "Los Angeles Kings" "Montreal Canadiens"
[9] "New Jersey Devils" "New York Islanders"
[11] "New York Rangers" "Ottawa Senators"
[13] "Philadelphia Flyers" "Phoenix Coyotes"
[15] "Pittsburgh Penguins" "San Jose Sharks"
[17] "St. Louis Blues" "Tampa Bay Lightning"
[19] "Toronto Maple Leafs" "Vancouver Canucks"
[21] "Washington Capitals" "Mighty Ducks of Anaheim"
[23] "Boston Bruins" "Buffalo Sabres"
[25] "Calgary Flames"
# Get the year datateamYears = fullPage %>%html_elements(cssYear) %>%html_text2()teamYears
# now combine the values in a dataframedfTeamInfo =data.frame (name=teamNames, year=teamYears)dfTeamInfo
name year
1 Colorado Avalanche 1996
2 Dallas Stars 1996
3 Detroit Red Wings 1996
4 Edmonton Oilers 1996
5 Florida Panthers 1996
6 Hartford Whalers 1996
7 Los Angeles Kings 1996
8 Montreal Canadiens 1996
9 New Jersey Devils 1996
10 New York Islanders 1996
11 New York Rangers 1996
12 Ottawa Senators 1996
13 Philadelphia Flyers 1996
14 Phoenix Coyotes 1996
15 Pittsburgh Penguins 1996
16 San Jose Sharks 1996
17 St. Louis Blues 1996
18 Tampa Bay Lightning 1996
19 Toronto Maple Leafs 1996
20 Vancouver Canucks 1996
21 Washington Capitals 1996
22 Mighty Ducks of Anaheim 1997
23 Boston Bruins 1997
24 Buffalo Sabres 1997
25 Calgary Flames 1997
26.13 Put it into a function
#-----------------------------------------------# Let's see if we can get the first page of data#-----------------------------------------------# The following function will be useful to get rid of extra "whitespace"# if necessary.removeWhitespace =function(text){ text =gsub("\\n","", text) text =gsub("\\t","", text )return ( gsub(" ", "", text))}x =" this is some info \n\nanother line\nanother line\tafter a tab"cat(x)
this is some info
another line
another line after a tab
# Create a function to scrape the hocky data from one of the # pages as specified in the url argument.getHockeyData <-function(url) { the_full_html =read_html(url) teamNames = the_full_html %>%# Analyze the HTML from one of the pages to figure out which CSS selector# is best to use. We did so and figured out that the hockey team names# were surrounded with an HTML that had class="name". Therefore the# best css slector to use was ".name"html_elements( ".name" ) %>%html_text() teamNames =removeWhitespace(teamNames) # the team names seem to be surrounded by whitespace wins = the_full_html %>%# Analyze the HTML from one of the pages to figure out which CSS selector# is best to use. We did so and figured out that the number of wins# were surrounded with an HTML that had class="wins". Therefore the# best css slector to use was ".wins"html_elements( ".wins" ) %>%# analyze the HTML to find the appropriate css selectorhtml_text() wins =removeWhitespace(wins)# ... we can keep doing this to get all the other data on the page # for each team ... We didn't do that here but feel free to fill in # the missing code to scrape the rest of the data from the webpages.return(data.frame(team=teamNames, wins=wins, stringsAsFactors=FALSE)) }
26.14 Some URLs includes info about the page number
#--------------------------------------------------------------------# Let's figure out how to write a loop to get multiple pages of data#--------------------------------------------------------------------# The following is the url without the page numberbaseurl ="https://scrapethissite.com/pages/forms/?page_num="# get data for the first pagepageNum ="1"url =paste0(baseurl, pageNum)getHockeyData (url)
# We can now write a function to get the data from multiple pages of the hockey data.# The pages argument is expected to be a vector with the numbers of # the pages you want to retrieve.getMultiplePages <-function(pages =1:10){# This is the URL without the page number baseurl ="https://scrapethissite.com/pages/forms/?page_num="# baseurl = "https://scrapethissite.com/pages/forms/?page_num=THE_PAGE_NUMBER&league=american# baseurl = "https://finance.yahoo.com/quote/<<<TICKER>>>?p=<<<TICKER>>>&.tsrc=fin-srch" allData =NA# Loop through all the pagesfor ( page in pages ){# Create the URL for the current page url =paste0(baseurl, page)# Get the data for the current page hd =getHockeyData(url)# Combine the data for the current page with all of the data# from the pages we have already retreived.if (!is.data.frame(allData)){# This will only happen for the first page of retrieved data. allData = hd } else {# This will happen for all pages other than the first page retrieved.# rbind will only work if allData is already a dataframe. Therefore# we cannot use rbind for the first page of retrieved data. allData =rbind(allData, hd) }# We don't want to overwhelm the web server with too many requests# so we will pause (i.e. sleep) for 1 second after every time we# retrieve a page before getting the next page of data.Sys.sleep(1) }return(allData)}getMultiplePages(4:6)
# BE CAREFUL - the next line may take a little time to run# allPages = getMultiplePages(1:24)## Let's try it with just 2 pages for now ...allPages =getMultiplePages(1:2)
26.16 Scrape the number of pages to get
# Figure out automatically how many pages of data there are on the websiteurl="https://www.scrapethissite.com/pages/forms/?page_num=1"fullPage =read_html(url)# css that targets the 24 that represents the 24th page of datacss ="li:nth-child(24) a"css ="li:last-child a"css =".pagination-area li:nth-last-child(2) a"fullPage %>%html_elements(css) %>%html_text2()
[1] "24"
26.17 robots.txt file
#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@# DEFINITION: root directory (or root folder) ##### # The "root" folder is the top level folder# on a computer harddrive or on a website.# It is named "/" on Mac and "\" on Windows.#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@# EXAMPLEdir("/") # show the files and folders in the root of the harddrive
# robots.txt ##### # Websites may place a file named robots.txt# at the very top of their website.## Examples# https://finance.yahoo.com/robots.txt # pretty simple# https://www.amazon.com/robots.txt # very complex# https://www.gutenberg.org/robots.txt # pretty typical### Example entries in robots.txt# Disallow all robots to scrape the entire site# Since Disallow specifies the "root" or top level folder## User-agent: *# Disallow: /## Allow all robots to scrape the entire site# since "Disallow: " specifies no path## User-agent: *# Disallow: ## Sleep 10 seconds between each page request## crawl-delay: 10## More info about robots.txt## Quick overview and a "test tool"# https://en.ryte.com/free-tools/robots-txt/## Ultimate robots.txt guide# https://yoast.com/ultimate-guide-robots-txt/## Another test tool# https://technicalseo.com/tools/robots-txt/#@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
26.18 Techniques for finding CSS selectors
############################################################################.# Techniques for finding CSS selectors## 1. Analyze the entire HTML code.# Right-click on blank area of page and choose "view page source"# (or something similar)## 2. Use the built in developer tools in most browsers.# Right click on the data you want to scrape and choose "inspect".# That brings you to the browser's developers tools. # Right click on the HTML Tag you want to get a CSS selector for# and choose "Copy selector". Then paste the selector into # a text editor (e.g. VSCode or RStudio's editor).# # Warning that these selectors will be too overly specific and# do not generalize well to scrape the same type of data from# the entire webpage.## For example the following css selector was retrieved in this way.# It gets a very specific position on the page. However, if the page# code changes in the future, this selector may stop to work. # This selector is very "fragile":## css selector to find the last page # #hockey > div > div.row.pagination-area > div.col-md-10.text-center > ul > li:nth-child(24) > a## 3. Use the "selector gadget" Chrome extension. # This is very helpful but is also not 100% guaranteed to work # correctly.############################################################################.
26.19 “scrape” the total number of pages.
############################################################################## Write code to "scrape" the total number of pages.## Find out how many pages there are (since the website may change, # we can't assume that there will always be the same number of pages)#############################################################################..................................................................# Find the CSS selector that pinpoints the number of the last page#..................................................................# Modern browsers have "web developer tools" built into them.# These tools can help you to find the css selector that you need.# Different browsers have slightly different tools. # For the following we will be using Chrome.## Navigate to the first page ( https://scrapethissite.com/pages/forms/?page_num=1 )## Right click on the last page number and choose "inspect". This will# reveal a new window that shows the HTML for the page. The part of the HTML# that corresponds to the page number is revealed. Make sure the page number# is there ...## Right click on the highlighted HTML and choose the menu choices :# "Copy | copy selector". This will copy into the computer's clipboard # a CSS selector that will pinpoint that piece of info.# # Paste the CSS selector into your R code. The following is the # css selector that I got when I did this:## #hockey > div > div.row.pagination-area > div.col-md-10.text-center > ul > li:nth-child(24) > a## Note that this css selector is "fragile" - i.e. it is likely not to work# in the future if more data is added to the stie. Therefore it is NOT a very # good CSS selector to use. We will examine this issue below. For now, let's # just use this selector and revisit this issue later.url ="https://scrapethissite.com/pages/forms/?page_num=1"cssSelector ="#hockey > div > div.row.pagination-area > div.col-md-10.text-center > ul > li:nth-child(24) > a"# save the html for the page in a variable so we can use it # again later without needing to pull it down again from the website.full_page =read_html(url) # Get last page number by searching the full_page html for the tags# identified by the cssSelector and stripping out the tagslast_page_number <- full_page %>%html_elements( cssSelector ) %>%html_text()last_page_number
[1] "\n \n 24\n \n "
# Strip out the whitespace last_page_number =removeWhitespace(last_page_number)last_page_number
[1] "24"
# Convert the page nubmer to a numeric valuelast_page_number =as.numeric(last_page_number)last_page_number
[1] 24
# Now that it works, let's create a function that packages up this functionality.# It's always a good idea to create a function that neatly packages up your# working functionality.## The function will take the "parsed" html code as an argument.getLastPageNumber <-function(theFullHtml) { cssSelector ="#hockey > div > div.row.pagination-area > div.col-md-10.text-center > ul > li:nth-child(24) > a"# remember that a function that does not have a return statement# will return the last value that is calculated. theFullHtml %>%html_elements( cssSelector ) %>%html_text() %>%removeWhitespace() %>%as.numeric()}last_page_number =getLastPageNumber(full_page)last_page_number
[1] 24
26.20 Timing R code with Sys.time()
# Now that we have the total nubmer of pages, we can get ALL of the data# for all the pages as follows.## NOTE that this may take a while to run.# We can figure out how much time by running all of the following commands# together. Sys.time() # shows the time on your computer
[1] "2024-04-16 12:47:28 EDT"
start =Sys.time()# put some r code hereend =Sys.time()end - start
Time difference of 0.0007381439 secs
start_time =Sys.time() # get the time before we startdataFromAllPages =getMultiplePages(1:last_page_number) # get all the pagesend_time =Sys.time() # get the time when it endedend_time - start_time # show how long it took
Time difference of 30.53477 secs
# Let's examine the results ...nrow(dataFromAllPages) # how many rows?
26.20.1 Finding a css selector that isn’t “fragile”
######################################################################### Finding css selectors that aren't "fragile"#########################################################################--------------------------------------------# Quick review of CSS selector rules. #--------------------------------------------# To review the CSS selector rules see the website # https://flukeout.github.io/ ## Read the tutorial information as well as try to solve the different # levels of the game. # The answers to the "game" on the above website can be found here:# https://gist.github.com/humbertodias/b878772e823fd9863a1e4c2415a3f7b6##-----------------------------------------------# Analyzing the CSS Selector we got from Chrome# (There is an issue with it. # We will address the issue and fix it.) #------------------------------------------------# The following was the selector that we got from Chrome # to find the last page number:## #hockey > div > div.row.pagination-area > div.col-md-10.text-center > ul > li:nth-child(24) > a## This selector is "fragile". It will work for now but might not work in the future. # This website has data for several years up until the present time. We # anticipate that in the future there will be additional pages of data.# The CSS selector that we found will work to find the 24th page number# (which is currently the last page). However, in the future if there are # more pages, the selector will continue to return the number 24. # To understand exactly why, let's analyze this selector:# For a review of the rules for how selectors are built see the following # page: https://flukeout.github.io/## <div> # div is the parent of h1 and ul# <h1>My stuff</h1> # h1 is the child of div and the sibling of ul# <ul> # ul is the parent of 3 <li>s, the 2nd child of div and a sibling of h1# <li> # # <strong> # Table# </strong></li># <li>Chairs</li># <li>Fork</li># </ul># </div># The following breaks down the selector that Chrome figured out into its # different parts:## #hockey # find the HTML start tag whose id has a value id="hockey" # > # directly inside of that element# div # there is a <div> tag# > # directly inside of that div tag# div.row.pagination-area # there is a <div> tag that has row and pagination-area classes, i.e. <div class="row pagination-area"># > # directly inside of that# div.col-md-10.text-center # there is div tag with classes "col-md-10" and "text-center", i.e. <div class="col-md-10 text-center"># > # directly inside of that # ul # there is a ul tag# > # directly inside that there is# li:nth-child(24) # an <li> tag that is the 24th tag inside of the enclosing <ul> tag# > # directly inside that there is # a # an "a" tag## You should take the time to convince yourself that the css selector is # accurate today by looking carefully at the HTML code and noting exactly where the # last page number is in the context of the other tags and attributes on the page. #................................................# Using VSCode (or another text ediotr) to analyze the HTML#................................................# It's much easier to navigate around the HTML code by analyzing the HTML in an HTML editor. # VSCode works well for this. To get the full HTML, right click on the page# and choose "View page source" (in Chrome or a similar link in other browsers).# Then copy all the code (ctrl-a on windows or cmd-a on Mac) and paste# it into a new VSCode file. Then save the file with a .html extension# to ensure that VSCode knows how to navigate around the file. # In VSCode you can now point to the left margin and press the arrows that appear# to "collapse" or "expand" the HTML tags. Doing so helps a lot in trying to # understand how the HTML file is organized.## Other useful features in VSCode to help with editing HTML:## - alt-shift-F # # Remember that HTML does not require proper indentation to work in the webpage.# However, without proper indentation, it is hard to read the HTML. Pressing# shift-alt-F will automatically "format" the entire HTML file so that it # is indented properly and is easier to read.## - As noted above - point to left margin to see arrows to # collapse or expand HTML elements.## - ctrl-shift-P## VSCode has many features that are not directly available through the# menus. ctrl-shift-P (Windows) or cmd-shift-P (mac) reveals the# "command palette". This is a quick way to search for commands# in VSCode which you might not otherwise know about. Try it. # For example,## * press ctrl-shift-P and type "comment" (without the quotes). Then choose# "Add line comment" to automatically insert <!-- --> into your HTML file.# You can add comments to the HTML file as you are examining it. This may# help you when you are trying to figure out the structure of a # complex HTML file and how to # ## * Highlight some text, press ctrl-shift-P. # Then type "case" (without the quotes). You will see options to transform # the selected text upper or lowercase. ## - For more tips on how to use VSCode to edit HTML files: # https://code.visualstudio.com/docs/languages/html## - For more tips on how to use VSCode in general:# https://code.visualstudio.com/docs/getstarted/tips-and-tricks#................................................# ... back to analyzing the CSS selector#................................................# From analyzing this HTML, we determine that each page number is in an <li># element. Since there are 24 pages of data there are 24 <li> tags (each contains# a link to one of the page numbers). There is also one more <li> tag that# contains the "next page" button. ## In the CSS Selector tha we got from Chrome "li:nth-child(24)" is specifically# targeting the 24th <li> tag. That is the last tag today. However, # if in the future more pages of data are added to the website, this selector# will still retrieve the 24th <li> tag and not the 2nd to last <li> tag# (remember, the last <li> tag is for the "next page" button).## (NOTE: the "next page" link appears in the HTML code# as "next page" but is rendered in the browser as ">>". Replacing the# words "next page" in the HTML with ">>" on the rendered page# is accomplished through CSS rules - but this has no effect on # what we're trying to accomplish. When scraping a page, you work with what's # in the HTML file, not what is rendered in the browser.)## Therefore it is important to analyze the HTML and try # to find a "less fragile" css selector the LAST page number the same information.# After looking at the HTML code carefully, it seems like the <ul> tag# referenced in the CSS selector that we found has a class="pagination" attribute, # ie. <ul class="pagination">. We can target that directly with ".pagination".## The li that points to the last page happens to be the 2nd to last # <li> tag that is directly inside this ul tag, i.e. ## <ul class="pagination"># <li> info about page 1 </li># <li> info about page 2 </li># etc ...# <li> info about the last page </li> # THIS IS THE <li> WE WANT# <li> info about the "next" button </li># </ul>## Our css selector therefore becomes : ".pagination > li:nth-last-child(2)"# Let's see if our code works with the new CSS selectorgetLastPageNumber <-function(theFullHtml) {# Selector specfies the # 2nd to last li inside the a tag that contains class="pagination" selector =".pagination > li:nth-last-child(2)"# remember that a function that does not have a return statement# will return the last value that is calculated. theFullHtml %>%html_elements( selector ) %>%html_text() %>%removeWhitespace() %>%as.numeric()}full_page =read_html("https://scrapethissite.com/pages/forms/?page_num=1")getLastPageNumber(full_page) # test the new version
