lucenedotnetnukedotnetnuke-9

DotNetNuke Lucene Search not working 'Lock obtain timed out' in load balanced env, how to fix?

We have a DotNetNuke site running on two servers that are load balanced. To ensure the files are in sync on these servers, we are using File Replication Service.

Search works fine on DotNetNuke when not load balanced, but in the load balanced setup the search stops working after a while (no suggestions, no results).
The following related exception is all over our log files:

[D:2][T:31][ERROR] DotNetNuke.Services.Exceptions.Exceptions - Lucene.Net.Store.LockObtainFailedException: Lock obtain timed out: NativeFSLock@D:\Sites\SiteName\App_Data\Search\write.lock
   at Lucene.Net.Store.Lock.Obtain(Int64 lockWaitTimeout)
   at Lucene.Net.Index.IndexWriter.Init(Directory d, Analyzer a, Boolean create, IndexDeletionPolicy deletionPolicy, Int32 maxFieldLength, IndexingChain indexingChain, IndexCommit commit)
   at Lucene.Net.Index.IndexWriter..ctor(Directory d, Analyzer a, MaxFieldLength mfl)
   at DotNetNuke.Services.Search.Internals.LuceneControllerImpl.get_Writer()
   at DotNetNuke.Services.Search.Internals.LuceneControllerImpl.Delete(Query query)
   at DotNetNuke.Services.Search.Internals.InternalSearchControllerImpl.DeleteSearchDocumentInternal(SearchDocument searchDocument, Boolean autoCommit)
   at DotNetNuke.Services.Search.Internals.InternalSearchControllerImpl.DeleteSearchDocumentsByModule(Int32 portalId, Int32 moduleId, Int32 moduleDefId)
   at DotNetNuke.Services.Search.SearchDataStore.StoreSearchItems(SearchItemInfoCollection searchItems)
   at DotNetNuke.Services.Search.SearchEngine.IndexContent()
   at DotNetNuke.Services.Search.SearchEngineScheduler.DoWork()

My best guess is that the issue is caused because both servers are running their search functionality, and the File Replication Service is syncing the files which causes conflicts.
What would be the best way to solve this?

Installation details:
DNN v. 09.02.00 (366)
.NET Framework 4.6

Solution

  • There's a 'scheduler' tool inside of the settings section that contains all CRON/background jobs functionality.

    DNN Scheduler Tool

    One of the background jobs is the 'Search: Site Crawler' job which is responsible for indexing the website. When that job runs at the same time on both servers, unexpected conflicts occur. To prevent this from happening, you can configure the job to only run on a specified server using the 'Servers' setting.

    Search: Site Crawler DNN Background Job

    After configuring the job to only run on one server, the issue did not come back and search still works on both servers.

    Thanks @Sanjay for pointing me in the right direction.