[SOLVED] Trying to scrape a PDF in R, my code will only scrape 6 out of 9 pages and i'm not sure why, am I missing something in my code?

Trying to scrape a PDF in R, my code will only scrape 6 out of 9 pages and i'm not sure why, am I missing something in my code?

Im trying to scrape a couple PDFs in R, PDF1 has 9 pages and PDF2 has 12 pages. When I run the code below it scrapes both PDFs but only up to page 6 and nothing after that. Is there a reason for this? Something missing in my code?

library(tm)
read <- readPDF(engine = "xpdf", control = list(text = "-layout")) 
document <- Corpus(URISource("C:\\Users\\Goku\\Documents\\Python Scripts\\PDF Scraping\\123.pdf"), readerControl = list(reader = read))
doc <- content(document[[1]])
head(doc)

You can find the pdf at: https://www.scribd.com/document/396797318/123

Solution

I can't replicate your issue. Using your document I get 12 pages reading the text in both ways. Checking if they are identical also yields true.

tm with reader pdftools:

library(tm)

read <- readPDF(engine = "pdftools", control = list(text = "-layout")) 
document <- Corpus(URISource("396797318-123.pdf"), readerControl = list(reader = read))

using pdftools directly:

library(pdftools)

text <- pdf_text("396797318-123.pdf")

Check if they are identical:

tm_text <- as.vector(sapply(document, as.character))
identical(text, tm_text)
[1] TRUE

str(document)
List of 1
 $ 396797318-123.pdf:List of 2
  ..$ content: chr [1:12] "            Training and Development Policy\r\n                                                     Contents\r\"| __truncated__ "1.2 As a guiding principle, all staff must be offered appropriate and relevant\r\n     development opportunitie"| __truncated__ "   • Short Term Externally Provided Training (e.g. one or two day external\r\n     courses)\r\n   • Longer Term"| __truncated__ "     are at no cost) together with periodic training bulletins informing employees of\r\n     impending courses"| __truncated__ ...
  ..$ meta   :List of 7
  .. ..$ author       : chr "leanne.cutts"
  .. ..$ datetimestamp: POSIXct[1:1], format: "2014-02-28 17:57:34"
  .. ..$ description  : chr ""
  .. ..$ heading      : chr "Training and Development Policy"
  .. ..$ id           : chr "396797318-123.pdf"
  .. ..$ language     : chr "en"
  .. ..$ origin       : chr "PDFCreator Version 1.4.3"
  .. ..- attr(*, "class")= chr "TextDocumentMeta"
  ..- attr(*, "class")= chr [1:2] "PlainTextDocument" "TextDocument"
 - attr(*, "class")= chr [1:2] "VCorpus" "Corpus"