I have been running a HttpWebRequest on a public website to return JSon for at least a couple of years. Last week it suddenly stopped working and would not give a response via this code (it is called Old because I am using a workaround with a powershell script, but it is a clumsy solution). It eventually times out after the default 100 seconds but a longer timeout would not fix it.
Function CreateRequestOld(ByVal sMESUser As String, ByVal sMeeting As String, ByVal sRaceDate As String, ByVal iRaceNumber As Integer, ByVal iTrade As Integer, Optional ByVal sJuristriction As String = "VIC") As String
Dim Url As String
Dim request As HttpWebRequest = Nothing
Dim dataStream As Stream = Nothing
Dim response As HttpWebResponse = Nothing
Dim strResponseStatus As String = ""
Dim reader As StreamReader = Nothing
Dim responseFromServer As String = ""
Dim sFile As String = "C:\logs\MESRaceLog_" & sMESUser & ".txt"
Dim Util As New Utilities
Dim sMeetingCode As String
Dim RI As New RaceInfo
Dim dte As New DateClass
Dim dtRaceDate As Date
If (iTrade = 1) Then
sFile = "C:\logs\MESRaceTradeLog_" & sMESUser & ".txt"
End If
If (iTrade = 2) Then
sFile = "C:\logs\MESRaceOddsLog_" & sMESUser & ".txt"
End If
Try
sMeetingCode = RI.GetTABCouseCode(sMeeting)
If (sMeetingCode = "") Then
Util.LogMsg(sFile, "TAB CreateRequest Error no meeting code for Meeting name " & sMeeting)
Return ""
End If
dtRaceDate = dte.ConvertStringToDate(sRaceDate, "dd/mm/yyyy")
sRaceDate = dte.Get_Date_String(dtRaceDate, "yyyy-mm-dd")
Url = "https://api.beta.tab.com.au/v1/tab-info-service/racing/dates/" & sRaceDate & "/meetings/R/" & sMeetingCode & "/races/" & iRaceNumber & "?jurisdiction=" & sJuristriction
Util.LogMsg(sFile, "TAB CreateRequest URL " & Url)
request = HttpWebRequest.Create(New Uri(Url))
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
request.Method = "GET"
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
request.AllowAutoRedirect = True
response = request.GetResponse()
strResponseStatus = CType(response, HttpWebResponse).StatusDescription
dataStream = response.GetResponseStream()
reader = New StreamReader(dataStream)
responseFromServer = reader.ReadToEnd()
reader.Close()
dataStream.Close()
response.Close()
Catch ex As Exception
Util.LogMsg(sFile, "CreateRequest Error" & vbCrLf & ex.Message)
End Try
Return responseFromServer
End Function
However, if I paste the URL into a browser (Chrome in my case) it responds with Json as expected.
So I tried Postman and it also does not respond if it is just the plain URL. If however, I copy the curl script and use Postman it does respond correctly. I notice with the curl script it has 23 request header elements, whereas with the plain URL (the one that fails) there are only 5 header elements.
I suspect that the website has changed to HTTP/2 and therefore requires all these extra header elements. So I copied the URL as Windows Powershell from Chrome (under developer tools, right click, copy as Windows Powershee) and created a script which is below and this runs well and grabs the response.
$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
$session.Cookies.Add((New-Object System.Net.Cookie("_cs_c", "0", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_tgpc", "d077be37-9bfe-58ce-bb34-2219fe18534c", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("s_ecid", "MCMID%7C42816005962630671342879997247751714175", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_gcl_au", "1.1.1461016396.1701314243", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("__qca", "P0-115178265-1701314243095", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_fbp", "fb.2.1701314243684.710804252", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_scid", "173c7c5f-d2aa-4407-9b6d-5eb48d7d5326", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_scid_r", "173c7c5f-d2aa-4407-9b6d-5eb48d7d5326", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_sctr", "1%7C1706792400000", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("at_check", "true", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("AMCVS_5C750C3A53DB4D070A490D4D%40AdobeOrg", "1", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_abck", "E98C9DC841935F14412438E52AEE528E~0~YAAQBcfOF59EHWuNAQAAsVHf0gvjwuRSy0ElwTKYqvAoJLQPTUBCXrzWbnUSm592eN9RKleGSHb6N1G4sOpkqms4YYyspcf+3j26XCyZ5cUO/PtysyIwLocjO/mDFZ0aQTYZIcMXSuuNycWI0uarkBykoIE5TBfeo6kPD1jD7LecoSHOPVRT/z/FC/0XUS8QcgfqHESSh7Gk6D8Dh3OBQ67qpu6GXq2enMxvn/8v0vn9u0DfH739OwrQyIfO82L11CRPswZtzXj0Wk5Qx5z9ObdAZ+eM9Edz+ts4SjQVXm585PwP9aam/xPL6Lqhov+8+htIFbaKT9O9WoemaBfthkvFk90cFfZ4tv57iPDPwT7+dIDGIcwph6wK5gsd9DS4kCpPXiOIgDcJIV4r6/tL2h8xgm+Ui+Sx~-1~-1~-1", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("bm_sz", "55EEBE064EF9FD3569F82C5AE9083857~YAAQBcfOF6FEHWuNAQAAsVHf0had2HtUmx7KB2ZnSoNh2rj59HNbyA1TAzJCjLmz8aXSPpV1Q699Pfux/9L01gXAmWItcxkttXyzNTYJBwN5/0YXC7Z5KMVxPA4YnDI+DPEUDgbo9pls/PxE86j6qFbQsHPk5mjH5trCuBl4I6lLa4we6cHSZZ2D8Rb8hNSdi3hfkgNtXDISvW35NyOo3EMgBYkb3SZL5BgjWsI6D/Bw9rpj80MRJs/NPZPHOaCvj76NQEHEr2FNclSS1j5sKa3dCeLQC3CfFtw705FlSHX+fxLCaJr0k+V3guclD9JQxD64qzawCSqOur3H8sGNdd6eRHPULHeijC+naKplTbnYgTtwUX2KdjE=~3621429~4469301", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("AKA_A2", "A", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("ak_bmsc", "F23866DFAFE8CC0C588C07C641D011C3~000000000000000000000000000000~YAAQOU5haKnFDcqNAQAAbNln0xZzvuAFiiDMoeSqPqgw/Qyd1vMSeWW1tBvtNwt1/1xz7GqrMaHlb6o/7sgtgmAm+DDYW43r4aLL0IpxZ2znjHkqMe9WUQGFW1z66zd5FWnbVJph+HPgdO38rC0bnaxlGp4EjsgLMVSp3JoveyRNFernNgur79/hEqApGaNw0wIdq+0uT1GIN7MzFitWaLYEXdbwgWmGYroXCzL4f2Jcxuz4jlsrvo+BhP36wRHKjmflq36j+D9bcg1E6QLT94auind7POTHnvh/uERvVKxGJdW6b4DpuHYiWo8s0EwK2PE2d3+7nXCH1iTxV7+2sDPmcMmqrPbGcrIH0UXR5bb7d2P64v4CgiynMUNx3AlK9JJBSm1czBQFKlMQ", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("www.tab.com.au", "179643557%7CMCIDTS%7C19777%7CMCMID%7C42816005962630671342879997247751714175%7CMCAAMLH-1709253617%7C8%7CMCAAMB-1709253617%7CRKhpRz8krg2tLO6pguXWp5olkAcUniQYPHaMWWgdJ3xzPWQmdj0y%7CMCOPTOUT-1708656017s%7CNONE%7CMCAID%7CNONE%7CMCCIDH%7C-394740951%7CvVersion%7C5.5.0", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("bm_mi", "7E78507F65544A6BEB911B6594252269~YAAQOU5haLLFDcqNAQAA1N1n0xaR8VACVmX0VDfQyabr26S83xcjsuZDySCB0Ba3KF8V7EBTrDO2I0q0LqCGVCXBIEpm4kqRUrsWAnZ2ku6pa3LWedJ/oPEPdqXXQz4MRQoqBv3xk/iKpQsEscblrzmKbEh1S+MoRjUsDv05k7V8/YypSTJlyfrhrA4XFBRJcyKThMUSgVmKmftm6lFi2Iv5X32BWk9tdClyDo1WcoR1gf3benLG+0dVsRJlJi3GhxxJh6NzUnbl5iuFh+p7VQtjevNJYtVX66OX773tSVIteUnl/rYsf5u/6dPm1F8FllaIkx0BtA==~1", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_cs_mk_aa", "0.1392149211434872_1708648818354", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("adcloud", "{%22_les_v%22:%22y%2Ctab.com.au%2C1708650618%22}", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_tguatd", "eyJzYyI6IihkaXJlY3QpIn0=", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_tgidts", "eyJzaCI6ImQ0MWQ4Y2Q5OGYwMGIyMDRlOTgwMDk5OGVjZjg0MjdlIiwiY2kiOiJkODhmOTQ2ZS0xNGY2LTU2MDUtODJlOS02N2FiZjA0NDQ0MzIiLCJzaSI6IjBhYzA0OGMzLTliZjgtNTBkZC04ZmZhLTQzY2U3N2ZkNGQ5MCJ9", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_tglksd", "eyJzIjoiMGFjMDQ4YzMtOWJmOC01MGRkLThmZmEtNDNjZTc3ZmQ0ZDkwIiwic3QiOjE3MDg2NDg4MTg1MTUsInNvZCI6IihkaXJlY3QpIiwic29kdCI6MTcwMTMxNDI0MjY2NSwic29kcyI6Im8iLCJzb2RzdCI6MTcwMTMxNDI0MjY2NX0=", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_gid", "GA1.3.286042675.1708648819", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_ga", "GA1.3.1084240241.1701314243", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_tgsid", "eyJscGQiOiJ7XCJscHVcIjpcImh0dHBzOi8vd3d3LnRhYi5jb20uYXUlMkZcIixcImxwdFwiOlwiT25saW5lJTIwU3BvcnRzJTIwJTI2JTIwSG9yc2UlMjBSYWNpbmclMjBCZXR0aW5nJTIwQXVzdHJhbGlhJTIwJTdDJTIwVEFCXCIsXCJscHJcIjpcIlwifSIsInBzIjoiZjk5OWNmMDktNDlkOC00OTM3LTgyMmQtNzVkYWJhMzlhYjM1IiwicHZjIjoiMSIsInNjIjoiMGFjMDQ4YzMtOWJmOC01MGRkLThmZmEtNDNjZTc3ZmQ0ZDkwOi0xIiwiZWMiOiI0IiwicHYiOiIxIiwidGltIjoiMGFjMDQ4YzMtOWJmOC01MGRkLThmZmEtNDNjZTc3ZmQ0ZDkwOjE3MDg2NDg4MjE4MDU6LTEifQ==", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_cs_id", "8c885f2f-4ec7-a81f-a51d-0c7fbc4e5253.1701314242.13.1708649137.1708648818.1.1735478242634.1", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_cs_s", "4.0.0.1708650937050", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("mbox", "PC#4d529f94d644416b83c7f3f4420ea6c1.36_0#1771893939|session#e85cb33e18f1458fae30265336746b93#1708650999", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_uetsid", "1e537e60d1e411eeb9d3b9dc7b7b2ef3", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_uetvid", "fa7b46408f2e11ee94a259059b2087fb", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("bm_sv", "FA3401F77595CB3BFA48E9B3EB68D923~YAAQB1rWF90I/s+NAQAAvYR40xY/4Sg4442TcILWMORP16M+CUypxE/1Q8GoIPyAaR3Dw6QFQYepBzb6OzgAuHqjwVwgKffGPGlNSIakay0I67nwTvg22/n4RLbWrXHiKR1WUnyy0xS1ldQdlsQLEU9TbDrhJbIKxcuaoIvFSlWb13RSwFgigd6I77vhvfog4dYam2Prx93Z9S5AsP/Yt8mjF+UnQjukoloUsVSrB/tmd1rbngxGNFwKDA08xL7MVjU=~1", "/", ".tab.com.au")))
$session.Cookies.Add((New-Object System.Net.Cookie("_ga_ZMHBYSQ22N", "GS1.1.1708648818.12.1.1708649915.60.0.0", "/", ".tab.com.au")))
Invoke-WebRequest -UseBasicParsing -Uri "https://api.beta.tab.com.au/v1/tab-info-service/racing/dates/ThisRaceDate/meetings/R/ThisRaceCode/races/999?jurisdiction=VIC" `
-WebSession $session `
-Headers @{
"authority"="api.beta.tab.com.au"
"method"="GET"
"path"="/v1/tab-info-service/racing/dates/today/meetings?jurisdiction=VIC"
"scheme"="https"
"accept"="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
"accept-encoding"="gzip, deflate, br"
"accept-language"="en-US,en;q=0.9"
"cache-control"="max-age=0"
"dnt"="1"
"if-none-match"="W/`"850d3337bda07909d530fde6a483487e`""
"sec-ch-ua"="`"Not A(Brand`";v=`"99`", `"Google Chrome`";v=`"121`", `"Chromium`";v=`"121`""
"sec-ch-ua-mobile"="?0"
"sec-ch-ua-platform"="`"Windows`""
"sec-fetch-dest"="document"
"sec-fetch-mode"="navigate"
"sec-fetch-site"="none"
"sec-fetch-user"="?1"
"upgrade-insecure-requests"="1"
} -OutFile Race.json
My question is how can I decorate the code HTTPWebRequest with all these 30 odd elements which are set with Powershell using the session.Cookies.Add command within code, and appear as items in the request header when run from a browser (or as a curl script in Postman).
I have tried adding a CookieContainer in my code but to no avail but maybe I didn't implement it properly. I never needed to set cookies in the past so my knowledge of how to implement them is limited.
I don't need a solution necessarily in VB.Net. I can change to C#. I only have VB.Net because of some legacy code and have not spent the time changing it all to C#
Update 1 I can now get this code to get a rsponse by adding a bunch of headers, the same as I find in the Powershell command.
This is now my code
'Using request As HttpWebRequest
request = HttpWebRequest.Create(New Uri(Url))
request.Pipelined = True
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
request.Method = "GET"
request.Timeout = 5000
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
request.Headers.Add("authority", "api.beta.tab.com.au")
request.Headers.Add("method", "GET")
request.Headers.Add("scheme", "https")
request.Headers.Add("accept-encoding", "gzip, deflate, br, zstd")
request.Headers.Add("accept-language", "en-US,en;q=0.9")
request.Headers.Add("cache-control", "max-age=0")
request.Headers.Add("dnt", "1")
request.AutomaticDecompression = DecompressionMethods.GZip
request.CookieContainer = cookies
response = request.GetResponse()
strResponseStatus = CType(response, HttpWebResponse).StatusDescription
dataStream = response.GetResponseStream()
reader = New StreamReader(dataStream)
responseFromServer = reader.ReadToEnd()
reader.Close()
dataStream.Close()
response.Close()
This will run successfully and return a response the first time, but I have this in a loop every 15 seconds. Sometimes the next loop will also work, but sometimes it won't. Sometimes 3 loops in a row work and then it fails and the next loop works. There is no consistency.
What am I doing wrong? I close the read stream so I don't think it is that. I am using just a single instance of the HTTPWebRequest and pass the object to this code.
I agree it is not HTTP/2 like I first suspected. It isn;t the timeout set to 5 seconds becuase even at 120 seconds it times out. When it works it takes milliseconds.
When it fails I've noticed the Pipelined is set to false, hence why I set it to true, but that seems to make no difference.
After at least a couple of days trying different options I have found a solution. So I am answering my own question.
What I have discovered is that when you send a request you need to provide a cookie container to receive the cookies when you grab the response. You must then send across that cookie container for the subsequent requests.
I have my request code in a function (VB .Net) and I was setting my cookie container as a new object each time I called this function.
I modified the function to take a cookie container as a parameter and created just one cookie container outside my loop, passed that across to the function each time. The first tiome the function is called, it populates the empty cookie container and then subsequent calls use the populated container.
Here is the code
Function CreateRequestSimple(ByVal sURL As String, ByVal sAuthority As String, ByRef request As HttpWebRequest, ByRef cookies2 As CookieContainer) As String
Dim Url As String
Dim dataStream As Stream = Nothing
Dim response As HttpWebResponse = Nothing
Dim strResponseStatus As String = ""
Dim reader As StreamReader = Nothing
Dim responseFromServer As String = ""
Dim cookies As CookieContainer = New CookieContainer()
Dim expiredcookie As Boolean
Try
request = HttpWebRequest.Create(New Uri(sURL))
request.Pipelined = True 'I don't think this does anything
request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36"
request.Method = "GET"
request.Timeout = 5000
request.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
request.Headers.Add("authority", sAuthority)
request.Headers.Add("method", "GET")
request.Headers.Add("scheme", "https")
request.Headers.Add("accept-encoding", "gzip, deflate, br, zstd")
request.Headers.Add("accept-language", "en-US,en;q=0.9")
request.Headers.Add("cache-control", "max-age=0")
request.Headers.Add("dnt", "1")
request.AutomaticDecompression = DecompressionMethods.GZip
expiredcookie = IsAnyCookieExpired(cookies2, request.RequestUri)
If (expiredcookie) Then
request.CookieContainer = cookies
cookies2 = cookies
Else
request.CookieContainer = cookies2
End If
response = request.GetResponse()
strResponseStatus = CType(response, HttpWebResponse).StatusDescription
dataStream = response.GetResponseStream()
reader = New StreamReader(dataStream)
responseFromServer = reader.ReadToEnd()
reader.Close()
dataStream.Close()
response.Close()
Catch ex As Exception
Return ex.Message
End Try
Return responseFromServer
End Function
Function IsAnyCookieExpired(ByVal cookies As CookieContainer, ByVal uri As Uri) As Boolean
Dim cookiecoll As New CookieCollection
' cookiecoll = cookies.
For Each cookie As Cookie In cookies.GetCookies(uri)
If (cookie.Expired) Then
Return True
End If
Next
Return False
End Function
I don't think I need to check if the cookie is expired because I think the expiry gets updated each time I call this function.
I call the function like this (psuedo code, not my real code)
Dim request As HttpWebRequest = Nothing
Dim cookies As CookieContainer = New CookieContainer()
Dim sResponse As String = ""
Dim sURL As String = ""
Dim sAuthority As String = "auth.com.au"
For i = 0 To 100
sURL = "http://someurl.com/extra"
sResponse = CreateRequestSimple(sURL, sAuthority, request, cookies)
Next i
Thanks to jdweng for indicating that cookies were the primary issue.
Additionally, the site had changed to sending their response in GZip format, so I had to deal with that change as well. Once again thanks the jdweng for suggesting that issue