How can I read the response from a web request when the Status is not 200?

I am having difficulty getting the response text from a HTTP web request in vb.net when I get a web exception.

This is the code I am doing it with.

Try
            myWebResponse = CType(request.GetResponse(), HttpWebResponse)
            myStreamReader = New StreamReader(myWebResponse.GetResponseStream())

            ResponseText = myStreamReader.ReadToEnd
            If myWebResponse.StatusCode = HttpStatusCode.Accepted Or myWebResponse.StatusCode = 200 Then
                SendResult = True 'Sent 
                SendStatus = 1 'message sent successfully
                Try
                    Integer.TryParse(myWebResponse.Headers("Number-Of-MT-PDU"), num_MT_PDU)
                Catch ex As Exception
                End Try
            Else
                SendStatus = 2 'message processed but not sent successfully
            End If
        Catch e As WebException
            If (e.Status = WebExceptionStatus.ProtocolError) Then
                Dim response As WebResponse = e.Response
                Using (response)
                    Dim httpResponse As HttpWebResponse = CType(response, HttpWebResponse)
                    statusCode = httpResponse.StatusCode
                    Try
                        myStreamReader = New StreamReader(response.GetResponseStream())
                        Using (myStreamReader)
                            ResponseText = myStreamReader.ReadToEnd & "Status Description = " & HttpWebResponse.StatusDescription
                        End Using
                    Catch ex As Exception
                        Logger.LogError(Me, ex)
                    End Try
                End Using

Annoyingly, the API I am contacting uses a 404 as a valid response. If I put the request in a browser some message text will be displayed. I want to be able to use that text in my program. I can not simply use the error code to determine actions as I don't think I can differentiate between a valid 404 response and an actual error.

In the code this line

myWebResponse = CType(request.GetResponse(), HttpWebResponse)

throws an exception.

In the exception I can get the 404 code and the description but not the response stream. It is always null.

If I get a 200 response I get the text in the Response stream no problem.

In the web exception response object (in Visual Studios debugger) I have checked the headers and the object values and can't find the response text anywhere. If I stick the request URL in a browser I get response text back even though it is a 404.

The raw response in fiddler:

HTTP/1.1 404 Not Found Connection: close Content-Type: text/plain; charset=UTF-8 Content-Length: 35 "The response Message"

Any ideas on how I can get "The response Message" in my program? I have to use .Net on the server.

Thanks for any help anybody can give.

Solution

This LINQPad query works fine, dumping the HTML provided by my web server's "Not Found" error web page:

Dim rq = System.Net.WebRequest.Create(New Uri("http://localhost/test"))
Try 
  Dim rs = rq.GetResponse
  rs.Dump
Catch Ex As System.Net.WebException
 Dim rs = Ex.Response
 Call (New StreamReader(rs.GetResponseStream)).ReadToEnd.Dump
End Try

FYI Your code works for me, except the presumed typo re HttpWebResponse.StatusDescription (and commenting out "unrelated stuff"), again as a LINQPad query (in .NET 4.0):

Dim request = WebRequest.Create("http://localhost/test")
Dim myStreamReader As StreamReader
Dim SendStatus As Integer = -1
Dim statusCode As HttpStatusCode
Dim ResponseText As String
Try
    Dim myWebResponse = CType(request.GetResponse(), HttpWebResponse)
        myStreamReader = New StreamReader(myWebResponse.GetResponseStream())

        ResponseText = myStreamReader.ReadToEnd
        If myWebResponse.StatusCode = HttpStatusCode.Accepted Or myWebResponse.StatusCode = 200 Then
            'SendResult = True 'Sent 
            SendStatus = 1 'message sent successfully
            'Try
            '    Integer.TryParse(myWebResponse.Headers("Number-Of-MT-PDU"), num_MT_PDU)
            'Catch ex As Exception
            'End Try
        Else
            SendStatus = 2 'message processed but not sent successfully
        End If
    Catch e As WebException
        If (e.Status = WebExceptionStatus.ProtocolError) Then
            Dim response As WebResponse = e.Response
            Using (response)
                Dim httpResponse As HttpWebResponse = CType(response, HttpWebResponse)
                statusCode = httpResponse.StatusCode
                Try
                    myStreamReader = New StreamReader(response.GetResponseStream())
                    Using (myStreamReader)
                        ResponseText = myStreamReader.ReadToEnd & "Status Description = " & httpResponse.StatusDescription ' HttpWebResponse.StatusDescription
                    End Using
                Catch ex As Exception
                    'Logger.LogError(Me, ex)
                    ex.Dump("Exception")
                End Try
            End Using
      End If
End Try
ResponseText.Dump("ResponseText")

I have also confirmed the above code (with the inferred As clauses added and converting the .Dump calls to Console.WriteLine) works in .NET 2.0 with VB8.