Currently I use 'lucene' and 'elasticsearch', and have next problem. I need get stemmed form or lemma for diminutive word. For instance :
etc.
But I get next results :
Is there any way (not important ready to use library, any algorithm, approach etc.) to get root / original word form for diminutive word forms?
Target language : Russian. For example :
Thanks in advance!
Firstly, as a side note: What you're trying to do isn't typically called stemming or lemmatiziation.
Your first issue would be mapping the token observed (e.g. собачка) to its normalised form (e.g. собака)-- Naively, this could be done by creating a SynonymFilter
which uses a SynonymMap
mapping dimunitive forms to their canonical forms. However, you'll likely run into problems with any natural language because not all derivations are unambiguous: For example, in German, Mädel ('girl'/'lass') could be a diminutive form of Magd (an archaic word meaning 'young woman'/'maid') or of Made ('maggot').
One way of disambiguating these two forms would be to calculate the probability of each canonical form appearing in the given context (e.g. the history of the preceding n tokens) and then replacing the dimunitive form with the most probable canonical form (using a custom-made TokenFilter
to do so)-- See e.g. the Wikipedia entry for word-sense disambiguation for different approaches.