Handling Exceptions and 4xx Status Codes in Invoke-WebRequest

TL;DR

• Invoke-WebRequest throws an exception when receiving responses like 4xx
• In PowerShell 7, you can use the -SkipHttpErrorCheck parameter to prevent throwing an exception even when receiving such responses
• In PowerShell 5.1, you have no choice but to handle the exception

Introduction

PowerShell has a cmdlet called Invoke-WebRequest, which allows you to send HTTP requests similar to curl.

First, let's try sending an HTTP request like the following.

try {
  $response = Invoke-WebRequest -Uri "https://learn.microsoft.com/en-us/azure/expressroute/expressroute-faqs"
  $response.StatusCode
} catch {
  $_.exception
}

If it's an access to a correct URL, the response is stored in $response, and 200 (meaning OK in HTTP status code) is output.

> try {
>>   $response = Invoke-WebRequest -Uri "https://learn.microsoft.com/en-us/azure/expressroute/expressroute-faqs"
>>   $response.StatusCode
>> } catch {
>>   $_.exception
>> }
200

Next, let's change the URL slightly and try to access a page that doesn't exist.
It's a bit subtle, but I've changed expressroute-faqs to expressroute-faq.

try {
  $response = Invoke-WebRequest -Uri "https://learn.microsoft.com/en-us/azure/expressroute/expressroute-faq"
  $response.StatusCode
} catch {
  $_.exception
}

And then, would you look at that—an exception is thrown.

> try {
>>   $response = Invoke-WebRequest -Uri "https://learn.microsoft.com/en-us/azure/expressroute/expressroute-faq"
>>   $response.StatusCode
>> } catch {
>>   $_.exception
>> }

Response         : StatusCode: 404, ReasonPhrase: 'Not Found', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
                   {
                     Request-Context: appId=cid-v1:b1c5b6ea-7ff0-41d3-9862-84c5e1dc3be7
                     X-Datacenter: wus
                     X-Frame-Options: SAMEORIGIN
                     X-Content-Type-Options: nosniff
                     X-UA-Compatible: IE=edge
                     X-XSS-Protection: 1; mode=block
                     X-Rendering-Stack: Dynamic
                     nel: {"report_to":"network-errors","max_age":604800,"success_fraction":0.01,"failure_fraction":1.0}
                     report-to: {"group":"network-errors","max_age":604800,"endpoints":[{"url":"https://mdec.nelreports.net/api/report?cat=mdocs"}]}
                     X-Azure-Ref: 0ohTbZQAAAADdBV7XAaZaSp4kbLw3grWOVFlPMDFFREdFMjMxOAA3MTY4OTIwZS05ZjViLTRhNjItYjE2ZS1kNWJlNjNjZTYxZTc=
                     Cache-Control: public, max-age=600
                     Date: Sun, 25 Feb 2024 10:21:22 GMT
                     Connection: keep-alive
                     Akamai-Cache-Status: Miss from child, Miss from parent
                     Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
                     Content-Type: text/html
                     Content-Length: 26839
                     Expires: Sun, 25 Feb 2024 10:31:22 GMT
                   }
HttpRequestError : Unknown
StatusCode       : NotFound
TargetSite       : Void ThrowTerminatingError(System.Management.Automation.ErrorRecord)
Message          : Response status code does not indicate success: 404 (Not Found).
Data             : {}
InnerException   :
HelpLink         :
Source           : System.Management.Automation
HResult          : -2146233088
StackTrace       :    at System.Management.Automation.MshCommandRuntime.ThrowTerminatingError(ErrorRecord errorRecord)

Meaning of 4xx status codes and how to handle them

Receiving a 404 (Not Found) response means that a response was successfully received from the server, and its content indicates that "the resource does not exist."
This implies that there are times when we don't want an exception to be thrown for such a case; rather, we want to receive that response and continue processing.
While we are using 404 (Not Found) as an example here, in the case of 429 (Too Many Requests), it is common to wait for the time specified in the response before retrying.

What to do

Invoke-WebRequest has a parameter called -SkipHttpErrorCheck. By using this, you can prevent exceptions from being thrown even when receiving responses such as 4xx.

> try {
>>   $response = Invoke-WebRequest -Uri "https://learn.microsoft.com/en-us/azure/expressroute/expressroute-faq" -SkipHttpErrorCheck
>>   $response.StatusCode
>> } catch {
>>   $_.exception
>> }
404

This allows you to perform actions such as judging that a resource doesn't exist and continuing other processing if it's a 404, or waiting for the time indicated in a header and retrying if it's a 429.

Personally, I was surprised to find that this wasn't implemented until PowerShell 7.

What to do in PowerShell 5.1

Since there's no choice but to handle the exception, it looks something like this. The status code is stored in a somewhat deep location: $_.Exception.Response.StatusCode.Value__.

$maxRetries = 10
$retryCount = 0

while ($retryCount -lt $maxRetries) {
    try {
        $response = Invoke-WebRequest -Uri $url -Method Post -Headers $headers -Body $body
        break
    } catch {
        # if return code is 429, wait for 10 seconds and retry
        if ($_.Exception.Response.StatusCode.Value__ -eq 429) {
            Start-Sleep -Seconds 10
            $retryCount++
        } else {
            throw $_
        }
    }
}

Summary

By default, Invoke-WebRequest throws an exception when it receives responses like 4xx. Therefore, you can use the -SkipHttpErrorCheck parameter to prevent it from throwing an exception even when receiving such responses.

References

• Invoke-WebRequest

https://learn.microsoft.com/powershell/module/microsoft.powershell.utility/invoke-webrequest#-skiphttperrorcheck?wt.mc_id=MVP_391314

• api - Invoke-Restmethod: how do I get the return code? - Stack Overflow
  It mentions that value__ is not a typo; it's easy to see why someone would think it is.

https://stackoverflow.com/questions/38622526/invoke-restmethod-how-do-i-get-the-return-code