rpdftmpdf-scrapingxpdf

Trying to scrape a PDF in R, my code will only scrape 6 out of 9 pages and i'm not sure why, am I missing something in my code?


Im trying to scrape a couple PDFs in R, PDF1 has 9 pages and PDF2 has 12 pages. When I run the code below it scrapes both PDFs but only up to page 6 and nothing after that. Is there a reason for this? Something missing in my code?

library(tm)
read <- readPDF(engine = "xpdf", control = list(text = "-layout")) 
document <- Corpus(URISource("C:\\Users\\Goku\\Documents\\Python Scripts\\PDF Scraping\\123.pdf"), readerControl = list(reader = read))
doc <- content(document[[1]])
head(doc)

You can find the pdf at: https://www.scribd.com/document/396797318/123


Solution

  • I can't replicate your issue. Using your document I get 12 pages reading the text in both ways. Checking if they are identical also yields true.

    tm with reader pdftools:

    library(tm)
    
    read <- readPDF(engine = "pdftools", control = list(text = "-layout")) 
    document <- Corpus(URISource("396797318-123.pdf"), readerControl = list(reader = read))
    

    using pdftools directly:

    library(pdftools)
    
    text <- pdf_text("396797318-123.pdf")
    

    Check if they are identical:

    tm_text <- as.vector(sapply(document, as.character))
    identical(text, tm_text)
    [1] TRUE
    
    str(document)
    List of 1
     $ 396797318-123.pdf:List of 2
      ..$ content: chr [1:12] "            Training and Development Policy\r\n                                                     Contents\r\"| __truncated__ "1.2 As a guiding principle, all staff must be offered appropriate and relevant\r\n     development opportunitie"| __truncated__ "   • Short Term Externally Provided Training (e.g. one or two day external\r\n     courses)\r\n   • Longer Term"| __truncated__ "     are at no cost) together with periodic training bulletins informing employees of\r\n     impending courses"| __truncated__ ...
      ..$ meta   :List of 7
      .. ..$ author       : chr "leanne.cutts"
      .. ..$ datetimestamp: POSIXct[1:1], format: "2014-02-28 17:57:34"
      .. ..$ description  : chr ""
      .. ..$ heading      : chr "Training and Development Policy"
      .. ..$ id           : chr "396797318-123.pdf"
      .. ..$ language     : chr "en"
      .. ..$ origin       : chr "PDFCreator Version 1.4.3"
      .. ..- attr(*, "class")= chr "TextDocumentMeta"
      ..- attr(*, "class")= chr [1:2] "PlainTextDocument" "TextDocument"
     - attr(*, "class")= chr [1:2] "VCorpus" "Corpus"