This is the outcome error and I can tell this is because there is at least one document without some term, but I don't get why and how I can solve it.
prep_fun = function(x) {
x %>%
str_to_lower %>% #make text lower case
str_replace_all("[^[:alpha:]]", " ") %>% #remove non-alpha symbols - chao punctuation y #
str_replace_all("\\s+", " ") %>% #collapse multiple spaces
str_replace_all("\\W*\\b\\w\\b\\W*", " ") #Remuevo letras individuales
}
tok_fun <- function(x) {
tokens <- word_tokenizer(x)
textstem::lemmatize_words(tokens)
}
it_patentes <- itoken(data$Abstract,
preprocessor = prep_fun,
tokenizer = tok_fun,
ids = data$id,
progressbar = F)
vocab <- create_vocabulary(it_patentes, ngram = c(ngram_min = 1L, ngram_max = 3L),
stopwords = tm::stopwords("english"))
pruned_vocab <- prune_vocabulary(vocab, term_count_min = max(vocab$term_count)*.01,
doc_proportion_min = 0.001)
vectorizer <- vocab_vectorizer(pruned_vocab)
dtm <- create_dtm(it_patentes, vectorizer,type = "dgTMatrix", progressbar = FALSE)
> #Plot the metrics to get number of topics
> t1 <- Sys.time()
> tunes <- FindTopicsNumber(
+ dtm = dtm,
+ topics = c(2:25),
+ metrics = c("Griffiths2004", "CaoJuan2009", "Arun2010"),
+ method = "Gibbs",
+ control = list(seed = 17),
+ mc.cores = 4L,
+ verbose = TRUE
+ )
fit models...Error in checkForRemoteErrors(val) :
4 nodes produced errors; first error: Each row of the input matrix needs to contain at least one non-zero entry
> print(difftime(Sys.time(), t1, units = 'sec'))
Time difference of 9.155343 secs
> FindTopicsNumber_plot(tunes)
Error in base::subset(values, select = 2:ncol(values)) :
object 'tunes' not found
Even though I know ldatuning is made for topicmodels, I don't think there might be a huge difference to get a number to start testing, is there?
ldatuning expects input dtm matrix in a different format (format from topicmodels package). You need to convert dtm (sparse matrix from Matrix package) to a format which ldatuning can understand