wcfbiztalkbiztalk-2013hl7-v2btahl7

Message Response Zombies occurring with errors Codes 0xC0C01B4C and 0xc0c016b5 no Orchestration

Consider the following message flow in BizTalk.

We have several MLLP receive ports/locations setup receiving HL7v2 messages in one application. These ports each receive slightly different message types.

Let's call this one RP1

In another application we have send ports that subscribe to each respective receive port. These send ports each have an outbound map that transforms the messages in HL7v3 and submits it to a WCF (request/response) service.

Let's call this SP1

The WCF service then processes and validates the HL7v3 and sends back an HL7v3 ack message. The SP1 send port has custom send and receive pipeline components. The receive (from the WCF response) just takes the message and promotes certain fields which will be used later for subscriptions.

There are then two more Send ports. SP2 that subscribes to positive ACKs. SP3 to the Negative based on the fields promoted above. The positive ACKs are just consumed and the negative ACKs are emailed off to support staff.

The issue is that in about 10% of messages we are seeing 1 of these 2 error messages popping up:

A response message for two-way receive port "SP.CDX.LAB_MICRO.SubmitCDA.WCFCustom" is being suspended as the messaging engine could not correlate the response to an existing request message. This usually happens when the host process has been recycled.
MessageId:  {731623F3-995B-4C57-BD21-12865AD78717}
InstanceID: {084BD473-C857-4C5E-A49B-8A86EA2CAC39}
The following stored procedure call failed: " { call [dbo].[bts_UpdateMsgbox_BizTalkServerReceive]( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)}". SQL Server returned error string: "The statement has been terminated.;Cannot insert duplicate key row in object 'dbo.InstancesSuspended' with unique index 'IX_InstancesSuspended_InstanceID'. The duplicate key value is (084bd473-c857-4c5e-a49b-8a86ea2cac39, afa466c7-3bd2-4cde-a293-3df3fb5d8543)."

Usually followed by a suspended service instance in the Group viewer:

 The instance completed without consuming all of its messages. The instance and its unconsumed messages have been suspended

The Service Name of the suspended instance is that of RP1. The Message type of the non-consumed message is that of the ACK from SP1 (so it's the WCF response). This is weird because in my mind RP1 should never be expecting this response message AND there are send ports (SP2, SP3) subscribed to the response message types.

The other point I forgot to make is, there are 3 Receive Ports like RP1, each with 3 Receive Locations and 3 Send Ports subscribing to their respective Receive Ports.

The BizTalk Server is installed across 2 physical servers sharing 1 BizTalkMgmtDb/Messagebox

Previous to this we had the same number of messages feeding in, but it was consolidated (at the sending end) to one single Receive Location. The old solution had multiple Orchestrations, but never experienced this issue.

So why now are the WCF (HL7v3) response messages getting lost and being suspended under an instance of RP1 (HL7v2)?

Here's a basic image of what it looks like.

biztalk layout

Solution

  • I was able to fix this by changing the value of the RouteDirectToTP value in the Context Properties of the WCF two-response message. Based on information in these posts:

    https://jeremiedevillard.wordpress.com/2010/01/11/how-work-request-response-pattern-part-3/

    https://bveldhoen.wordpress.com/2010/09/05/messaging-only-request-response-correlation/

    Apparently by default, BizTalk tries to route an ACK back to the original receive port by default. I just did this in my pipeline component:

     msgReceived.Context.Promote("RouteDirectToTP", "http://schemas.microsoft.com/BizTalk/2003/system-properties", false);

    and so far the problem has stopped. No more Zombies and no more errors regarding the stored procedure failing.