There is WCF Service (NET:TCP | Reliable | SecurityEnabled) running as console on Physical Machine A
There are 20-30 .NET client applications (Winforms) exposed over Citrix so hosted again on One physical machine different from server.
The application structure follows the same pattern as discussed in http://msdn.microsoft.com/en-us/magazine/cc163537.aspx by Juval Lowy.
The problem is at the server it is able to invoke the callback with no errors, but the client never receives it. There is code tries to subscribe again if no callback call within 60secs. It has a side effect that it opens a new connection by calling Subscribe API on the server. Over a period you can see many TCP connections open on the server. No errors but still client callback is never invoked.
Sometimes following error is thrown:
The message could not be transferred within the allotted timeout of 00:01:00. There was no space available in the reliable channel's transfer window. The time allotted to this operation may have been a portion of a longer timeout.
In addition MaxBufferPoolSize is set to Int64.MaxValue as you can see in the code below
var binding = new NetTcpBinding(SecurityMode.Transport, reliableSession);
binding.ReliableSession.Enabled = true;
binding.ReliableSession.InactivityTimeout = TimeSpan.FromDays(1);
binding.ReliableSession.Ordered = true;
binding.CloseTimeout = TimeSpan.FromHours(1);
binding.SendTimeout = TimeSpan.FromHours(1);
binding.ReceiveTimeout = TimeSpan.FromHours(1);
binding.OpenTimeout = TimeSpan.FromHours(1);
binding.ReaderQuotas.MaxDepth = Int32.MaxValue;
binding.ReaderQuotas.MaxStringContentLength = Int32.MaxValue;
binding.ReaderQuotas.MaxArrayLength = Int32.MaxValue;
binding.ReaderQuotas.MaxBytesPerRead = Int32.MaxValue;
binding.ReaderQuotas.MaxNameTableCharCount = Int32.MaxValue;
binding.MaxBufferPoolSize = Int64.MaxValue;
binding.MaxReceivedMessageSize = Int32.MaxValue;
Any suggestions will be great help!
I recently discovered the callback on the client side has different kinds of processing based on what is visible on the user screen. This was blocking the callbacks to be acknowledged. That explains the buffer overflow as well. The frequency of notifications sent from the server is very high compared to the time taken by each client to process them.
Apart from the changes I did like disabling security, configuring timeouts correctly, following line of code helped a lot:
public void OnNotification(AmigoMessage messsage)
{
ThreadPool.QueueUserWorkItem((x) => { ProcessNotification(messsage); });
}