rquantedadata-preprocessing

Backtransform word tokens to a sentence-based corpus in Quanteda after preprocessing

I want to preprocess my text data using the {quanteda} package in R. To do so, I am creating a corpus, which is then tokenized and preprocessed (e.g. lowercase, remove punctuation, etc.).

Ideally, I would then want to restore the initial sentence structure of the corpus, whilst keeping the document variables, because I am following a string-of-words approach in the analysis.

# Create an example corpus.
my_corpus <- corpus(c("This is a sentence. \n\nThis is another sentence.", 
                      "This is the first sentence of the second document.",
                      "This is yet another ... ••• *** sentence."))

# Set docvars.
docvars(my_corpus) <- data.frame(doc_id = 1:3, author = c("A", "B", "C"))

# Three documents and four sentences.
ndoc(my_corpus)
nsentence(my_corpus)

# Tokenize and preprocess.
my_tokens <- my_corpus %>%
  tokens(remove_punct = T) %>%
  tokens_tolower()

my_tokens

# Docvars are still present.
docvars(my_tokens)

I could then simply do the following to restore the sentence structure. However, in the process of doing so, I would lose my docvars:

# Back-transform to sentences.
my_corpus.clean <- vapply(my_tokens, paste, collapse = " ", character(1)) %>% corpus()

# Docvars are lost.
docvars(my_corpus.clean)

The preprocessing worked and so did restoring the sentence structure, but I no longer have my docvars. I could then add them back to the new corpus object (docvars(...) <- ...), but am afraid that the docvars values will no longer correspond to the right documents.

Is there a way to transform the tokens object back to a sentence-based object that avoids losing the docvars?

Solution

  • Try this at the end:

    # back-transform to sentences.
my_corpus.clean <- vapply(my_tokens, paste, collapse = " ", character(1)) |>
    corpus(docvars = docvars(my_tokens))

# docvars are present
my_corpus.clean
#> Corpus consisting of 3 documents and 2 docvars.
#> text1 :
#> "this is a sentence this is another sentence"
#> 
#> text2 :
#> "this is the first sentence of the second document"
#> 
#> text3 :
#> "this is yet another sentence"

docvars(my_corpus.clean)
#>   doc_id author
#> 1      1      A
#> 2      2      B
#> 3      3      C