I created a Lucene RAMDirectory to collect data from different sources and make them quickly searchable. I spent many hours to understand the different analyzers and index strategies, but in some cases the query result is not the expected.
Here is a demo class:
class LuceneDemo {
static final String ANIMAL = "animal";
static final String PERSON = "person";
private StandardAnalyzer analyzer = new StandardAnalyzer();
private IndexSearcher searcher;
private IndexWriter writer;
LuceneDemo() {
Directory ramDirectory = new RAMDirectory();
IndexWriterConfig config = new IndexWriterConfig(analyzer);
try {
writer = new IndexWriter(ramDirectory, config);
addDocument(createDocument(PERSON, "DR-(frankenstein)"));
addDocument(createDocument(ANIMAL, "gray fox"));
addDocument(createDocument(ANIMAL, "island fox"));
writer.close();
IndexReader reader = DirectoryReader.open(ramDirectory);
searcher = new IndexSearcher(reader);
} catch (IOException e) {
e.printStackTrace();
}
}
private Document createDocument(String type, String value) {
Document document = new Document();
document.add(new TextField("type", type, Field.Store.YES));
document.add(new TextField("name", value, Field.Store.YES));
document.add(new StringField("name", value, Field.Store.YES));
return document;
}
private void addDocument(Document document) {
try {
writer.addDocument(document);
} catch (IOException e) {
e.printStackTrace();
}
}
List<String> getDocuments(String type, String value) {
value = "*" + QueryParser.escape(value) + "*";
try {
QueryParser queryParser = new QueryParser("name", analyzer);
queryParser.setAllowLeadingWildcard(true);
queryParser.setDefaultOperator(QueryParser.Operator.AND);
BooleanQuery.Builder query = new BooleanQuery.Builder();
query.add(new TermQuery(new Term("type", type)), BooleanClause.Occur.MUST);
query.add(queryParser.parse(value), BooleanClause.Occur.MUST);
TopDocs docs = searcher.search(query.build(), 10);
return Arrays.stream(docs.scoreDocs).map(scoreDoc -> {
try {
return searcher.doc(scoreDoc.doc).get("name");
} catch (IOException e) {
return "";
}
}).collect(Collectors.toList());
} catch (ParseException | IOException e) {
e.printStackTrace();
}
return Collections.emptyList();
}
}
If I search for "ox", "gray fox" or "-(frankenstein)", the code works pretty well. But I have no result for "DR-(frankenstein)". I have no idea what I did wrong. So any suggestions are welcome.
// OK
luceneDemo.getDocuments(LuceneDemo.ANIMAL, "ox").forEach(System.out::println);
luceneDemo.getDocuments(LuceneDemo.ANIMAL, "gray fox").forEach(System.out::println);
luceneDemo.getDocuments(LuceneDemo.PERSON, "-(frankenstein)").forEach(System.out::println);
// NOT OK
luceneDemo.getDocuments(LuceneDemo.PERSON, "DR-(frankenstein)").forEach(System.out::println);
This is how your documents are indexed -
Basically StringField
indexes field irrespective of the analyzer
- without tokenizing, and lowering the case. Whereas the reader is using StandardAnalyzer
and lowers the case for all search.
Hence, searching for "DR-(frankenstein)" searches for "dr-(frankenstein)" which doesn't have a match.
To make your code work using StandardAnalyzer you need to index StringField as lowercase.
document.add(new StringField("name", value.toLowerCase(), Field.Store.YES));