rtopic-modelingtopicmodelsquanteda

Quanteda with topicmodels: removed stopwords appear in results (Chinese)

My code:

library(quanteda)
library(topicmodels)

# Some raw text as a vector
postText <- c("普京 称 俄罗斯 未 乌克兰 施压 来自 头 条 新闻", "长期 电脑 前进 食 致癌 环球网 报道 乌克兰 学者 认为 电脑 前进 食 会 引发 癌症 等 病症 电磁 辐射 作用 电脑 旁 水 食物 会 逐渐 变质 有害 物质 累积 尽管 人体 短期 内 会 感到 适 会 渐渐 引发 出 癌症 阿尔茨海默 式 症 帕金森 症 等 兔子", "全 木 手表 乌克兰 木匠 瓦列里·达内维奇 木头 制作 手表 共计 154 手工 零部件 唯一 一个 非 木制 零件 金属 弹簧 驱动 指针 运行 其他 零部件 材料 取自 桦树 苹果树 杏树 坚果树 竹子 黄杨树 愈疮木 非洲 红木 总共 耗时 7 打造 手表 不仅 能够 正常 运行 天 时间 误差 保持 5 分钟 之内 ")

# Create a corpus of the posts
postCorpus <- corpus(postText) 

# Make a dfm, removing numbers and punctuation
myDocTermMat <- dfm(postCorpus, stem = FALSE, removeNumbers = TRUE, removeTwitter = TRUE, removePunct = TRUE)

# Estimate a LDA Topic Model 
if (require(topicmodels)) {
  myLDAfit <- LDA(convert(myDocTermMat, to = "topicmodels"), k = 2)
}

terms(myLDAfit, 11)

The code works and I see a result. Here is an example of the output:

    Topic 1  Topic 2 
 [1,] "木"     "会"    
 [2,] "手表"   "电脑"  
 [3,] "零"     "乌克兰"
 [4,] "部件"   "前进"  
 [5,] "运行"   "食"    
 [6,] "乌克兰" "引发"  
 [7,] "内"     "癌症"  
 [8,] "全"     "等"    
 [9,] "木匠"   "症"    
[10,] "瓦"     "普"    
[11,] "列"     "京"

Here is the problem. All of my posts have been segmented (necessary pre-processing step for Chinese) and had stop words removed. Nonetheless, the topic model returns topics containing single-character stop terms that have already been removed. If I open the raw .txt files and do ctrl-f for a given single-character stop word, no results are returned. But those terms show up in the returned topics from the R code, perhaps because the individual characters occur as part of other multi-character words. E.g. 就 is a preposition treated as a stop word, but 成就 means "success."

Related to this, certain terms are split. For example, one of the events I am examining contains references to Russian president Putin ("普京"). In the topic model results, however, I see separate term entries for "普" and "京" and no entries for "普京". (See lines 10 and 11 in output topic 2, compared to the first word in the raw text.)

Is there an additional tokenization step occurring here?

Edit: Modified to make reproducible. For some reason it wouldn't let me post until I also deleted my introductory paragraph.

Solution

  • Here's a workaround, based on using a faster but "dumber" word tokeniser based on space ("\\s") splitting:

    # fails
features(dfm(postText, verbose = FALSE))
## [1] "普"     "京"     "称"     "俄罗斯" "未"     "乌克兰" "施压"   "来自"   "头"     "条"     "新闻"  
# works
features(dfm(postText, what = "fasterword", verbose = FALSE))
## [1] "普京"   "称"     "俄罗斯" "未"     "乌克兰" "施压"   "来自"   "头"     "条"     "新闻"

    So add what = "fasterword" to the dfm() call and you will get this as a result, where Putin ("普京") is not split.

    terms(myLDAfit, 11)
##      Topic 1  Topic 2         
##  [1,] "会"     "手表"          
##  [2,] "电脑"   "零部件"        
##  [3,] "乌克兰" "运行"          
##  [4,] "前进"   "乌克兰"        
##  [5,] "食"     "全"            
##  [6,] "引发"   "木"            
##  [7,] "癌症"   "木匠"          
##  [8,] "等"     "瓦列里达内维奇"
##  [9,] "症"     "木头"          
## [10,] "普京"   "制作"          
## [11,] "称"     "共计"

    This is an interesting case of where quanteda's default tokeniser, built on the definition of stringi's definition of text boundaries (see stri_split_boundaries, does not work in the default setting. It might after experimentation with locale, but these are not currently options that can be passed to quanteda::tokenize(), which dfm() calls.

    Please file this as an issue at https://github.com/kbenoit/quanteda/issues and I'll try to get working on a better solution using the "smarter" word tokeniser.