Update 2022-08-12

I re-thought about it and realized I was overcomplicating it. I found the best way to enhance this system is by using good old information retrieval techniques ie using 'location' of a word in a sentence and 'ranking' queries to display best hits. The approach is illustrated in this following picture.

---

Update 2015-10-15

Back in 2012, I was building a personal online application and actually wanted to re-invent the wheel because am curious by nature, for learning purposes and to enhance my algorithm and architecture skills. I could have used apache lucene and others, however as I mentioned I decided to build my own mini search engine.

Question: So is there really no way to enhance this architecture except by using available services like elasticsearch, lucene and others?

---

Original question

I am developing a web application, in which users search for specific titles (say for example : book x, book y, etc..) , which data is in a relational database (MySQL).

I am following the principle that each record that was fetched from the db, is cached in memory , so that the app has less calls to the database.

I have developed my own mini search engine , with the following architecture:

This is how it works:

• a) User searches a record name
• b) The system check what character the query starts with, checks if query there : get record. If not there, adds it and get all matching records from database using two ways:

• Either query already there in the Table "Queries" (which is a sort of history table) thus get record based on IDs (Fast performance)
  
• Or, otherwise using Mysql LIKE %% statement to get records/ids (Also then keep the used query by the user in history table Queries along with the ids it maps to).
  -->Then It adds records and their ids to the cache and Only the ids to the inverted index map.
  • c) results are returned to the UI

The system works fine, however I have Two main issues, that i couldn't find a good solution for (been trying for the past month):

First issue:
if you check point (b) , case where no query "history" is found and it has to use the Like %% statement : this process becomes time consuming when the query matches numerous records in the database (instead of one or two):

• It will take some time to get records from Mysql (this is why i used INDEXES on the specific columns)
• Then time to save query history
• Then time to add records/ids to cache and inverted index maps

Second issue:
The application allows users to add themselves new records, that can immediately be used by other users logged in the to application.
However to achieve this, inverted index map and table "queries" have to be updated so that in case any old query matches to the new word. For example if a new record "woodX" is being added, still the old query "wood" does map to it. So in order to re-hook query "wood" to this new record, here is what i am doing now:

• new record "woodX" gets added to "records" table
• then i run a Like %% statement to see which already existing query in table "queries" does map to this record(for example "wood"), then add this query with the new record id as a new row: [ wood, new id].
• Then in memory, update inverted index Map's "wood" key's value (ie the list), by adding the new record Id to this list

--> Thus now if a remote user searches "wood" it will get from memory : wood and woodX

The Issue here is also time consumption. Matching all query histories (in table queries) with the newly added word takes a lot of time (the more matching queries, the more time). Then the in memory update also takes a lot of time.

What i am thinking of doing to fix this time issue, is to return the desired results to the user first , then let the application POST an ajax call with the required data to achieve all these UPDATE tasks. But i am not sure if this is a bad practice or an unprofessional way of doing things?
So for the past month ( a bit more) i tried to think of the best optimization/modification/update for this architecture, but I am not an expert in the document retrieval field (actually its my first mini search engine ever built).

I would appreciate any feedback or guidance on what i should do to be able to achieve this kind of architecture.
Thanks in advance.

PS:

• Its a j2ee application using servlets.
• I am using MySQL innodb (thus i cannot use full-text search option)

---

I would strongly recommend Sphinx Search Server, wchich is best optimized in full-text searching. Visit http://sphinxsearch.com/.

It's designed to work with MySQL, so it's an addition to Your current workspace.