Intro

Whenever I look at the front page of Project Euclid, they try to be helpful and show me the top articles of the week, but it’s not exactly what I want to see. I want to know the all-time top articles from AOS, and I don’t just want a list of 5 articles, I want all the data.

True, there are some papers and blog posts out there that do some tabulation of the top articles in the whole discipline of statistics, but I want a break-down for a specific journal, and I want to use the most up to date information I can find. I also want to be able to see the top articles by keyword in each journal¹, and by decade, so we’re just going to have roll up our sleeves and do this the hard way :)

Why would someone want this information? It’s just intellectual curiosity. Sometimes I flip through a journal, and I start reading, but it’s kinda aimless if the journal is too technical. I can read American Statistician cover to cover because they mix it up with lots of different themes. Not so much with AOS. I need some help, because I’m only going to read a few articles a year. So this project is effectively about making a recommendation engine for an academic journal.²

library(rvest) # Web scraping
library(rcrossref) # Get citations
library(data.table) # Data tables and disk I/O
library(magrittr) # this is not a pipe

Goal: We need to collect a list of all published articles in AOS, and get the number of citations for each of the articles. The plan is to use a couple web scrapers to get the job done.

Getting all the articles in AOS

First, crawl around the main AOS landing page. The AOS landing page can be changed to the landing page of any journal hosted on Project Euclid. Later, I’m going to want to adapt this code to work on some other journals, such as the Annals of Applied Statistics.

Let me be the first to admit that I’m not a web-scraping guru, so my scraping strategy is likely suboptimal. But ya gotta make some mistakes before you know how to avoid them. Let me know how I can make things more stable and faster for my next go at it!

## This is the landing page for AOS
url <- "https://projecteuclid.org/all/euclid.aos"
main <- read_html(url)

## Suck up all links from the main page
aos_links <- html_nodes(main, "a") %>% html_attr("href") # links

## regex to match issues from the main page.
link_to_issue <- grepl("euclid.aos/[[:digit:]]+", aos_links)

## Toss out the links that do not appear to correspond to any issues.
aos_links <- aos_links[link_to_issue]

Now we have links to all landing pages of each issue the journal has released. Each issue page contains links to the articles in that issue, and each article page contains the precious DOIs. We will use the DOIs to search for the number of citations later. That’s the goal.

Here is a function to loop through all the articles in an issue. The main idea is that the function takes in a prepared, but empty, data.table and goes through all the articles in issue_links, and spits out a completed data.table for that issue. Each row of the outputted data.table corresponds to a single article, and on each article we record the DOI, year, authors, and article name. Nice!

issue_data_table <- function(DT, issue_links) {

  ## This loop goes through all the articles in an issue, and gets all the article
  ## information
  for (i in seq_along(issue_links)) {

    ## Go each each article page.
    ## Each article page is an element in the vector of issue_links.
    article_url <- paste0("https://projecteuclid.org", issue_links[i], "#info")

    message("Downloading an article...")

    ## FIXME: Wrap this in TryCatch()
    article_html <- read_html(article_url)

    ## Get the DOI
    ## Finding the correct nodes takes some fiddling around with SelectorGadget
    DOI_unclean <- article_html %>%
      html_node("p:nth-child(5)") %>%
      html_text()
    ## Some of the article information needs to be cleaned up.
    ## I prefer base-R regex functions when the job is not too bad :)
    DOI_clean <- sub("^Digital Object Identifier", "", DOI_unclean)

    ## Get the year of each article
    year_unclean <- article_html %>%
      html_node("#info p:nth-child(2)") %>%
      html_text()
    year_clean <- sub("^Source", "", year_unclean)

    ## Get article name
    art_name <- article_html %>%
      html_node("h3") %>%
      html_text()

    ## Get authors
    authors <- article_html %>%
      html_node(".small") %>%
      html_text()

    ## FIXME: Additionally get the keywords for each article. I forgot to do
    ## that after starting the AOS crawler.

    ## Assign all the article information to one row of a pre-made data.table
    ## `set` is a loopable version of `:=`.
    set(DT, i, "DOI", DOI_clean)
    set(DT, i, "Year", year_clean)
    set(DT, i, "Article_name", art_name)
    set(DT, i, "Authors", authors)

    ## Pause for 10 secs to not overwhelm the server.
    Sys.sleep(10)
  }
  DT
}

Apply the issue -> data.table function to each issue of the AOS catalog, while taking care to fail gracefully.

get_aos <- function(aos_links) {
  for (i in seq_along(aos_links)) {
    ## Crawl on the main page for each issue.
    issue_url <- paste0("https://projecteuclid.org", aos_links[i])

    ## Make a different data.table for each issue and name them programmatically.
    DTname <- paste0("issueDT_",
                     regmatches(issue_url, regexpr("[[:digit:]]+", issue_url)))

    ## Filename for each data.table
    fname <- paste0("./assets/", DTname, ".csv")

    ## If we already have that data, next
    if (file.exists(fname)) {
      message(paste("Checked issue", issue_url, "   ", i / length(aos_links) * 100, "% Done."))
      next
    }

    ## Fail silently, if it must be ...
    issue_html <- tryCatch(read_html(issue_url), error = function(e) e)

    ## Skip this issue if there is something wrong, we can try again later.
    ## https://stackoverflow.com/a/8094059/7281549
    if (inherits(issue_html, "error")) {
      message(paste("Problem encountered with issue"), aos_links[i], "Trying the next issue...")
      next
    }

    ## Search the issue page for links to each paper
    issue_links <- html_nodes(issue_html, "a") %>% html_attr("href")
    link_to_papers <- grepl("^/euclid.aos/[[:digit:]]+", issue_links)
    issue_links <- issue_links[link_to_papers] %>% unique()

    ## Make empty data.table for the issue information with the right dimensions
    assign(DTname,
           data.table(DOI = character(length = length(issue_links)),
                      Year = character(length = length(issue_links)),
                      Article_name = character(length = length(issue_links)),
                      Authors = character(length = length(issue_links))))

    ## Use our previous function to get the issue information into the data.table
    assign(DTname, issue_data_table(get(DTname), issue_links))

    ## Save the data.table
    fwrite(get(DTname), fname)

    ## Update the R console with our progress.
    message(paste("Downloaded issue", issue_url, "   ", i / length(aos_links) * 100, "% Done."))

    ## Pause again, do not burden ProjectEuclid.
    Sys.sleep(10)
  }
}

Let it rip:

get_aos(aos_links)

Looping through all the articles in all the issues takes about a day, because we cannot go too quickly without getting IP-banned from the website. I found this out the hard way …

The outer loop output looks something like this, so we can keep track of progress and see any failures when they inevitably arise. Normally I would have used txtProgressBar(), but I do want to print out which issue has a failure on each iteration, so the progress bar is a bit superfluous.

Checked issue https://projecteuclid.org/euclid.aos/1176342455     98.8679245283019 % Done.
Checked issue https://projecteuclid.org/euclid.aos/1176342405     99.2452830188679 % Done.
Checked issue https://projecteuclid.org/euclid.aos/1176342358     99.622641509434 % Done.
Checked issue https://projecteuclid.org/euclid.aos/1193342377     100 % Done.
...

When there is a failure with data retrieval, I think I’ve got things rigged correctly to ensure that it happens before the data.table is written to disk, so re-running the code will skip all “correct” data.tables and just fill in the missing ones.

Fast-forward 12 hours of web crawling …

Now we have a bunch of data.tables, one for each issue, with all the information required to run a citation search against each entry. All the data.tables were saved to individual files, so that we don’t have to run the function again :)

clean_all_issues <- function() {

  ## get a list of all issues.
  issues <- list.files("./assets/", pattern = ".csv$")

  for (i in seq_along(issues)) {

    issue <- fread(paste0("./assets/", issues[i]))[grepl("^doi:", DOI)][
      !grepl("^Discussion", Article_name)][
        !grepl("^Rejoinder", Article_name)][
          !grepl("^Correction", Article_name)][
            !grepl("^Volume Information", Article_name)]

    fwrite(issue, paste0("./assets/cleaned_issues/", issues[i]))
  }
}

clean_all_issues()

Getting the citation data

append_citations <- function(DT) {
  ## Make some space to store the citations (if not there)
  ## We just make the citations a column of -1's initially
  ## We can't use 0, because the article may actually have 0 cites
  if (!("citations" %in% colnames(DT)))
    DT[, citations := rep(-1, .N)]

  ## Now loop through all articles in an issue.
  for (i in seq_len(nrow(DT))) {

    ## skip some rows that evaded my crappy data cleaning skills.
    if (!grepl("^doi:", DT[i, DOI])) next

    read_cites <- DT[i, citations]

    ## an article is already present, no need to try again.
    if ((read_cites > -1 | read_cites == "error")) {
      message(paste("Article", DT[i, DOI], "is already here!"))
      next
    }

    doi <- sub("^doi:", "", DT[i, DOI])

    ## Get the number of citations for the article
    article_citations <- cr_citation_count(doi)$count

    ## save the data
    if (is.numeric(article_citations)) {
      set(DT, as.integer(i), "citations", article_citations)
      message(paste("Just got article", DT[i, DOI], "... The count was", article_citations))
    } else {
      set(DT, as.integer(i), "citations", "error")
      message(paste("Error with article", DT[i, DOI]))
    }

    ## wait 10 seconds. Do not poke the bear.
    Sys.sleep(5)
  }
  message("Finished an issue!")
  DT
}

get_citations <- function(issues_list) {

  ## Loop over all the issues.
  for (i in seq_along(issues_list)) {
    issue <- issues_list[i]

    ## If the data.table has citation information, use that, otherwise get a clean data.table
    if (issue %in% list.files("./assets/cite_tables/")) {
      issue_DT <- fread(paste0("./assets/cite_tables/", issue))
    } else {
      issue_DT <- fread(paste0("./assets/cleaned_issues/", issue))
    }

    message("Now working on ", issue)

    ## Apply the citations retrieval function to the table
    issue_with_cites <- append_citations(issue_DT)

    ## save or overwrite.
    fwrite(issue_with_cites, paste0("./assets/cite_tables/", issue))
    message(i / length(issues_list) * 100, "% Done.")
  }
}

if (!dir.exists("./assets/cite_tables")) dir.create("./assets/cite_tables")

issues <- list.files("./assets/cleaned_issues/", pattern = ".csv$")
get_citations(issues)

...
Just got article doi:10.1214/12-AOS1025 ... The count was 7
Just got article doi:10.1214/12-AOS1026 ... The count was 18
Finished an issue!
80% Done.
Now working on issueDT_1351602526.csv
Just got article doi:10.1214/11-AOS949 ... The count was 130
Just got article doi:10.1214/12-AOS999 ... The count was 45
...

Tabulate Results

First, gotta combine all the issue tables into one table. There were some parsing warnings that I’m going to ignore for now…

issues <- list.files(paste0("./assets/cite_tables/"), pattern = ".csv$", full.names = TRUE)
list_of_issues <- lapply(issues, fread)
issuesDT <- rbindlist(list_of_issues)[order(-citations)]

## Column 1 ['doi:10.1214/aos/1018031261'] of item 18 is missing in item 1. Use fill=TRUE to fill with NA (NULL for list columns), or use.names=FALSE to ignore column names. use.names='check' (default from v1.12.2) emits this message and proceeds as if use.names=FALSE for  backwards compatibility. See news item 5 in v1.12.2 for options to control this message.

issuesDT[, Article_name := gsub("\r?\n|\r", "", Article_name)]

summary(issuesDT$citations)

##     Min.  1st Qu.   Median     Mean  3rd Qu.     Max.     NA's 
##     0.00     6.00    17.00    56.84    43.00 22674.00       48

The top 100 AOS articles:

knitr::kable(head(issuesDT, 100), row.names = TRUE)