[SOLVED] System.Net.HttpListenerException: Failed to listen on prefix 'xxxx' because it conflicts with an existing registration on the machine

Whenever deploy our component we were receiving the following exception. Sometimes the component itself won’t start, sometimes the Service Console might say started but the service wont process any message. The error is pretty straightforward. There is a another HttpListener still listening to the prefix.

NServiceBus.Hosting.GenericHost System.Exception: Failed to start listener for http://localhost/products/ make sure that you have admin priviliges ---> System.Net.HttpListenerException: Failed to listen on prefix 'http://localhost:80/products/' because it conflicts with an existing registration on the machine.
   at System.Net.HttpListener.AddAllPrefixes()
   at System.Net.HttpListener.Start()
   at NServiceBus.Gateway.Channels.Http.HttpChannelReceiver.Start(String address, Int32 numWorkerThreads) in c:\BuildAgent\work\nsb.master_6\src\gateway\NServiceBus.Gateway\Channels\Http\HttpChannelReceiver.cs:line 30
   --- End of inner exception stack trace ---
   at NServiceBus.Gateway.Channels.Http.HttpChannelReceiver.Start(String address, Int32 numWorkerThreads) in c:\BuildAgent\work\nsb.master_6\src\gateway\NServiceBus.Gateway\Channels\Http\HttpChannelReceiver.cs:line 37
   at NServiceBus.Gateway.Receiving.GatewayReceiver.Start(Address localAddress) in c:\BuildAgent\work\nsb.master_6\src\gateway\NServiceBus.Gateway\Receiving\GatewayReceiver.cs:line 38
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at NServiceBus.Unicast.UnicastBus.NServiceBus.IStartableBus.Start(Action startupAction) in c:\BuildAgent\work\nsb.master_6\src\unicast\NServiceBus.Unicast\UnicastBus.cs:line 867
   at NServiceBus.Hosting.GenericHost.Start() in c:\BuildAgent\work\nsb.master_6\src\hosting\NServiceBus.Hosting\GenericHost.cs:line 34

When we perform the deep analysis.

This issue is only happening when you run the NServiceBus.Master Profile
This issue is only happening whenever your perform service restart from Service Console
There is no matter how many WorkerThreads you run

It puzzled us, when we read the NSB code much deeper, we found that HttpListener is closed during the object Dispose. Addition to this, when you stop the service the entire Process itself killed, so there will not be any object living on the machine to listen to the given URL prefix. BUT, we were wrong. When we analyze much closure, we found that the NServinceBus.Host process is still running even after Service-Console considered the service was stopped. Whenever we stop the Service on Service Console, the Process is still active on the machine & it took quite longer to shutdown completely. So, when we perform restart, there is a chance that there is a more than one instance of process for current service – one is trying to start another one is still trying to stop. We confirmed that the restart process throw the above exception.

We took the dump of the process which was shutting down. We found that, there are background threads still active on the process though the Service Console showing the status as stopped. There are lot of locks/ObjWait during the Dispose. For sure the stopped service is not yet closed the HttpListener object.

        Child SP               IP Call Site
000000001f19e048 00000000778818ca [GCFrame: 000000001f19e048] 
000000001f19e118 00000000778818ca [HelperMethodFrame_1OBJ: 000000001f19e118] System.Threading.Monitor.ObjWait(Boolean, Int32, System.Object)
000000001f19e230 000007fe99d57ccb System.Transactions.CommittableTransaction.Commit()
000000001f19e2d0 000007fe99d577df System.Transactions.TransactionScope.InternalDispose()
000000001f19e3c0 000007fe99d56f1c System.Transactions.TransactionScope.Dispose()
000000001f19e590 000007fe99c4a3b5 NServiceBus.Utils.TransactionWrapper.RunInTransaction(System.Action, System.Transactions.IsolationLevel, System.TimeSpan)
000000001f19e610 000007fe99c45b2b NServiceBus.Unicast.Transport.Transactional.TransactionalTransport.Process()
000000001f19e680 000007fe99c455cb NServiceBus.Utils.WorkerThread.Loop()
000000001f19e6d0 000007fef6fdf8a5 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000001f19e830 000007fef6fdf609 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000001f19e860 000007fef6fdf5c7 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
000000001f19e8b0 000007fef6ff2d21 System.Threading.ThreadHelper.ThreadStart()
000000001f19ebc8 000007fef81cf713 [GCFrame: 000000001f19ebc8] 
000000001f19eef8 000007fef81cf713 [DebuggerU2MCatchHandlerFrame: 000000001f19eef8] 
000000001f19f0d8 000007fef81cf713 [ContextTransitionFrame: 000000001f19f0d8] 
000000001f19f2c8 000007fef81cf713 [DebuggerU2MCatchHandlerFrame: 000000001f19f2c8]

We found that, the HttpListener was active & listening.

        Child SP               IP Call Site
000000002038e2f8 000000007788135a [InlinedCallFrame: 000000002038e2f8] System.Net.UnsafeNclNativeMethods+HttpApi.HttpReceiveHttpRequest(System.Runtime.InteropServices.CriticalHandle, UInt64, UInt32, HTTP_REQUEST*, UInt32, UInt32*, System.Threading.NativeOverlapped*)
000000002038e2f8 000007fe99c6a2f8 [InlinedCallFrame: 000000002038e2f8] System.Net.UnsafeNclNativeMethods+HttpApi.HttpReceiveHttpRequest(System.Runtime.InteropServices.CriticalHandle, UInt64, UInt32, HTTP_REQUEST*, UInt32, UInt32*, System.Threading.NativeOverlapped*)
000000002038e2b0 000007fe99c6a2f8 DomainBoundILStubClass.IL_STUB_PInvoke(System.Runtime.InteropServices.CriticalHandle, UInt64, UInt32, HTTP_REQUEST*, UInt32, UInt32*, System.Threading.NativeOverlapped*)
000000002038e3b0 000007fe99c697a4 System.Net.HttpListener.GetContext()
000000002038e4d0 000007fe99c6924f NServiceBus.Gateway.Channels.Http.HttpChannelReceiver.HttpServer()
000000002038e530 000007fef6fdf8a5 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000002038e690 000007fef6fdf609 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
000000002038e6c0 000007fef6fdf5c7 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
000000002038e710 000007fef6ff2d21 System.Threading.ThreadHelper.ThreadStart()
000000002038ea28 000007fef81cf713 [GCFrame: 000000002038ea28] 
000000002038ed58 000007fef81cf713 [DebuggerU2MCatchHandlerFrame: 000000002038ed58] 
000000002038ef38 000007fef81cf713 [ContextTransitionFrame: 000000002038ef38] 
000000002038f128 000007fef81cf713 [DebuggerU2MCatchHandlerFrame: 000000002038f128]

Question: Why the Service Console reports the service is stopped, even though the process is still active on the machine

NOTE: To reproduce this issue, you have to install your host as Windows Service using NServiceBus.Master profile and try to restart the service, you may need more attempts to get this issue.

Update:

When I read NSB code, on HttpChannelReceiver.Start(), its starting the Listener on the new Thread.

This listener thread is Foreground thread. (all other worker threads are BG thread)
No stop signal goes to this Foreground thread
There is no variable/fields holding the reference of this Thread - aborts are not possible.

This listener thread only reclaimed when ServiceHost kills the memory of current process, until ServicHost collects it, this thread will be blocked from exiting the current process.